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NEW ZEALAND PATENTS ACT 1953 



PROVISIONAL SPEdFtCATION 



A CHOICE-BASED METHOD FOR CALIBRATING ADDiriVE POINTS SYSTEM 



20 We, PAUL HANSEN, a New Zealand citizen, of 24 Maitiand Street, Dunedin New 
Zealand, and FRANZ OMBLER. a New Zealand citizen, of 19 Edinburgh Terrace, 
Berhampore. Wellington, New Zealand, do hereby declare this invention to be described 
in the following statement: 
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A CHOICE-BASED METHOD FOR CALIBRATING ADDITIVE 

POINTS SYSTEMS 

FIELD OF INVENTION 

The invention relates to additive points systems and in particular to a system, method and 
computer program for determining (calibrating) the points values of additive points 
systems. 

BACKGROUND TO THE INVENTION 

Additive points systems (APSs), a type of multi-attribute utility model or multiple criteria 
decision analysis tool, are also known as 'linear', 'point-count' and 'scoring' systems. 
APSs are widely and increasingly used worldwide in most branches of medicine for 
prioritisation, diagnosis and predictive purposes and in a wide range of other applications, 
including selecting immigrants, assessing mortgage applications and predicting parole 
violations, business bankrupteies and college graduations. They are also used as a generic 
project ^praisal tool. 

Canada and New Zealand, for example, recently developed APSs for a wide range of 
elective surgeries and other publicly-funded health cares. APSs are also used by the 
immigration systems of New Zealand, Canada, Australia and Germany and, to a lesser 
extait, the United Kingdom. 

APSs in general represent a relatively simple solution to the pervasive problem faced by 
decision makers with multiple criteria or attributes to consider, particularly when ranking 
alternatives or individuals. Hereinafter in this document alternatives and individuals are 
referred to generically as 'alternatives'. 

Specifically, APSs serve to combine alternatives' characteristics on multiple criteria to 
produce a single ranking of alternatives with respect to an over-arching criterion (such as, 
for example, the order in which to treat patients), or, more simply, to reach a decision (for 
example, whether or not to admit an immigrant). 
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la addition to their having been near universally found in many studies to be more 
accurate than 'expert' decision makers in the respective fields to which they have been 
appUed. the appeal of points systems is that liiey are simple to use. 

Each alternative's categorical rating on each criterion that is deemed relevant to the over- 
archmg cnterion is scored a particular number of points, that usually increase with the 
•importance' of the categories, and the points are summed (hence additive points systems) 
to produce a total score for the alternative. 

Alternatives are ranked with respect to the over-aiching criterion according to their total 
scores, mcludingbemgdeclinedaltogetherif a particular score 'threshold' is not reached 
UsuaUy the higher an alternative's score the higher its ranking, and the scores typicaUy 
nave no other meaning than this. 

Most APSs for elective surgeries and immigration respectively (the examples referred to 
^ove). have between five and seven criteria and two to five categories on each criterion 

Figures 1 ^^d 2 show an example of an APS used in Canada for prioritismg patients for 
mp or knee replacement surgery. 

In this example there are seven criteria, mostly based on tj^es of pain and functional 
hmxtations. Each criterion has a number of mutually exclusive and exhaustive categories 
on whxch the consulted decision maker (usually a doctor) is asked to rate the patient being 
considered. For example, item 2 in Figure 1, refers to the criterion of "Pain at Rest" and 
requires that the patient be assigned to one of four categories: •'None", "MUd" 
"Moderate" or "Severe". 



In general, if the criteria and the categories on each have been chosen for a particular 
APS system, then the point values for that APS must be determined (caUbrated) such that 
the resulting ranking of alternatives represents the decision maker.' preferences. TTxe 
invention is a new system, method and computer program for doing this. 
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In addition to the arbitrary assignment of points, there are two main existing q>proaches 
to calibrating APSs. 

5 The first regresses decision makers' judgements of the relative priorities or importance of 
a sample of real or hypothetical alternative ^profiles' (in other words, the alternative's 
categorical ratings on the criteria) on their characteristics in terms of the criteria and 
derives point values fi-om the regression coefficients. Usually only a small proportion of 
existing or theoretically possible profiles is surveyed because of the responder burden on 
10 the consulted decision makers. 

The above-mentioned decision makers' judgements are often elicited via a visual 
analogue scale (VAS), Item 8 of Figure 2 is such a VAS, where the decision maker is 
asked 'to rate the urgency or relative priority of this patient" between 'TSTot urgent at all" 
15 and "Extremely urgent (just short of an emergency)". This *score' and the patient's 
characteristics in terms of Items 1 to 7 of the same figure, along with analogous data for 
other patients, may then be used to calibrate the point values for the various criteria and 
categories using multiple regression techniques, as explained above. 

20 The second existing approach to calibration uses decision makers' judgements of the 
pairwise relative importance of the APS's criteria to derive ratio scale weights. These 
weights are then applied to normalised criteria values to derive point values. An example 
of this type of technique is the Analytic Hierarchical Process (AHP). 

25 Thus the first approach assumes that decision makers' judgements have interval scale 
measurement properties and the second ^proach assumes they have ratio scale 
properties. Both assumptions are relatively stringent and current techniques for eliciting 
decision makers' judgements have well-known biases. With respect to the first approach, 
for example, the validity of the dependent variable (the experts' judgments) and therefore 

30 the point values derived firom the estimated coefficients, can be criticised on two main 
groxmds. 
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First, the scaling methods such as the VAS used to elicit the experts' judgments of the 
profiles' relative priorities are based on mere introspection rather than the expression of a 
choice. Second. VAS in general may not actually have the scaling measurement 
5 properties required for the valuations that they produce to be inteipi^ted as relative 
pnorities rather tiian just as rankings. 

It is therefore desnable to have a method of calibrating new APSs or recalibrating or 
vahdating extant ones that require only ordinal measurement properties, specifically the 
10 positive expression of a ranking over paii^ of alternatives, as this is the least stringent of 
measurement property requirements. It would also be desirable for this method to achieve 
accurate results while reducing the burden on decision makers of ranking pairs of 
alternatives by minimising the number of pairs tiiey have to rank. 

15 SUMMARY OF THE INVENTION 

In broad tenns. in one form the invention provides a method for cah^rating additive 
points systems (APSs) comprising a pr^detennined pluraKty of criteria, each criterion 
havmg one or more categories wherein tiie points for each category of each criterion are 
determined by the pairwise ranking of profile pairs, each profile comprising a set of one 
or more of the criteria, each criterion in tiie set instantiated witii one of tiie categories for 
that criterion. 



Preferably the metiiod comprises tixe steps of generating ambiguous profile pairs for the 
APS to be caUbrated. resolving the ambiguous profile pairs, and solving tiie resulting 
25 system of equaUties/inequalities to obtain tiie point values. Ambiguous profile pairs are 
profile pairs in which one profile has a higher categorical rating on at least one criterion 
and a lower rating on at least one otiier criterion than tiie otiier profile. 

Preferably tiie step of generating flie ambiguous profile pairs comprises tiic fimher step of 
30 removmg any profiles tiiat are tiieoretically impossible. 
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Preferably the step of generating the ambiguous proffle paire comprises the further step of 
reducing all profile pairs in which both profiles have one or more of the same criteria 
instantiated with the same category. 

In broad terms in another form, the invention provides a system for calibrating additive- 
points systems (APSs) comprising a predetermined plurahty of criteria for the additive 
points system, each criterion capable of being instantiated with one or more predefined 
categories; and a points caUbrator configured to determme appropriate points for each 
category of each criterion by preparing data for and processing the results of the pairwise 
ranking of profile pairs, each profile comprising a set of one or more of the criteria, each 
criterion in the set instantiated with one of the categories for that criterion. 

Preferably the points caUbrator comprises an ambiguity generator configured to generate 
ambiguous profile pairs for the APS to be caUbrated. 

Preferably the points calibrator comprises a data input component configured to receive 
and store the equalities/inequalities that result fi-om resolving the ambiguous profile pairs 
generated by the ambiguity generator. 

Preferably the points calibrator comprises a solution component configured to solve the 
resulting system of equafities/inequalities to obtain the point values. 

Preferably the ambiguity generator is finther configured to remove any profiles that are 
theoretically impossible. 

Preferably tiie ambiguity generator is fiirther configured to reduce all profile pairs in 
which both profiles have one or more of the same criteria instantiated with the same 
category. 



In broad terms in yet another form the invention provides a conq)uter program for 
calibrating additive points systems (APSs) comprising an initialization component 
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configured to receive and store data representing a plurality of criteria for an APS and the 
categories with which each criterion may be instantiated, an ambiguity generator 
configured to generate ambiguous profile pairs for the APS to be caUbrated, a resolution 
component configured to select and present profile pairs to a user to be explicitly 
resolved and to store the results of the resolution, an ambiguity management component 
configured to manage the resolved and unresolved ambiguities and to automatically 
resolve any ambiguities that can be resolved impUcitly; and a solution component 
configured to solve the system of resolved inequaHties/equalities from the resolution 
component and the ambiguity management component. 

Preferably the computer program further comprises a revision component configured to 
allow a user to revise any resolved ambiguities. 

Preferably the computer program is implemented using linear programming. 
BRIEF DESCRIPTION OF THE FIGURES 

Preferred forms of the method system and computer program for calibrating additive 
points systems wiU now be described with reference to the accompanying figures in 
which: 

Figure 1 shows a prior art means of eliciting expert judgments for the purposes of 
caUbrating an APS via multiple regression-based techniques; 

Figure 2 shows a continuation of tiie prior art means of eliciting expert judgments for tiie 
25 purposes of caUbrating an APS via multimedia regression-based techniques; 

Figure 3 shows a preferred configuration of hardware for carrying out the invention; 



15 



20 



le m an 



Figure 4 shows a decision tree identifying flie 12 rankings of tiie 8 profiles possible 
30 exemplar APS with tiuree criteria and two categories and aUowing strict preferences only 
and no ties (for illustrative purposes only); 
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Figure 5 is a flow diagram Ulustrating the main steps in the method and computer 
program of the invention; 



are 



Figure 6 is a table showing the ambiguities from the six proffles (excluding those that 
unambiguous) for an exemplar APS with three criteria and two categories; 

Figure 7 is a table of the total and unique ambiguities for different APSs and degrees; 

Figure 8 is a continuation of the table fix,m Figure 7 of the total and unique ambiguities 
for different APSs and degrees; 

Figure 9 is a flow diagram Ulustrating the main components of the 'effici«it ambiguities 
generator' described in Step 1 of the method of the invention; 

Figure 10 is a table of the sufficient (but not necessary) conditions for implicitly 
resolving the B'^-degree ambiguities of an exemplar APS with three criteria and two 
categories; 

Figure 11 is a flow diagram illustrating an overview of Steps 1 to 5 of the computer 
program of Uie invention; 

Figure 12 is a flow diagram illustrating the preferred main components of the ambiguity 
generator of the computer program; 

Figure 13 is a flow diagram illustrating the prefeited main components involved in 
testing whether or not an ambiguity is resolved for the computer program; 

Figure 14 is a flow diagram illustrating the preferred main components involved in &e 
expUcit resolution of ambiguities for the computer program; and 
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Figure 15 is a flow diagram illustrating the preferred main components of the undo 
module (whereby expUcitly resolved ambiguities are revised) of the computer program. 

DETAILED DESCRIPTION OF THE PREFERRED FORMS 

The invention is primarily embodied in the methodology set out below both by itself and 
as implemented through computing resources such as the preferred resources set out in 
Figure 3, by way of example. The invention is also embodied in the software used to 
implement the methodology and in any system comprising a combination of hardware 
and software used to implement the methodology. 

hi its most preferred form the invention is implemented on a personal computer or 
workstation operating under the control of appropriate operating and application 
software. 



15 Figure 3 shows the preferred system architecture of a personal computer, workstation, or 
server on which the invention could be implemented. The computer system 300 typically 
comprises a central processor 302, a main memory 304, for example RAM, and an 
input/output controller 306. The computer system 300 may also comprise peripherals 
such as a keyboard 308, a pointing device 310. for example a mouse, tovclxpad, or 

20 trackball, a display or screen device 3 12, a mass storage memory 3 14, for example a hard 
disk, floppy disk or optical disc and an output device 316 such as a printer. The system 
300 could also include a networic interfece card or controller 318 and/or a modem 320. 
The individual components of the system 300 could communicate through a system bus 
322. 



25 



The method of the present mvention will now be described with reference to several 
examples, beginnmg with an APS with just three criteria: a, b and c; and two categories 
on each: 1 and 2; such that there are six criterion-category variables: al, a2, bl, b2, cl 
and c2. 



30 
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For example, if this were an APS for selecting imnugrants. criterion a might be 
•educational qualifications h 'wealth' and c 'language proficiency'. Category 1 might be 
geaerically defined as 'low' and 2 as 'high'. Real immigration APSs. however, typically 
have at least twice as many criteria and categories as this simple example. 

By definition, the values of these variables monotonically increase with the categories 
within each criterion so that the following 'inherent inequaUties' hold: «2 > a\, bl > b\ 

and c2> rA 



Corresponding to aU possible combinations of the two categories on the abc criteria, eight 
profiles, each with a total score equation, are represented by this system (where tiie three- 
digit numbers are symboUc representations only): 



222 = a2 + 62 + c2 
221 =a2 + fe2 + cl 
212 = a2 + 61 +c2 
122 = al +62 + C2 
112 = aH-fel +c2 
121 =al +fe2 + cl 
2n=a2 + b\ +cl 
111 =al +6H-C1 



The essence of calibrating an APS such as this one is deciding the point values of the six 
variables (al. «2. b\, bl, c\ and c2) such that decision makers' preferred (or 'vaUd') 
overall ranking of the eight profiles (equations) is reaUsed. 

The internal logic of APSs - specifically, the inviolable laws of arithmetic - restricts 
the otherwise 8! = 40.320 rankings (permutations), given strict preferences only and no 
ties (for illustrative pmposes only), to the 12 rankings represented via the decision tree in 
Figure 4. The decision tree highlights the inherent contingencies in the derivations of tiie 
profile's rankings. 

Any of tixe 12 rankings shown in Figure 4 can be produced from the six variables, 
depending on tiie values chosen for fliem. Ranking #1, for example, is given by al = 0, 
al = 4, 61 = 0, 62 = 2, cl = 0 and c2 = 1, with the total scores: 222 = 7, 221 = 6. 212 = 5, 
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211 = 4, 122 = 3, 121 = 2, 112 = 1 and 11 1 = 0. Alternatively, ranking #12, for example, 
is given by al = 0, a2 = 1, 61 = 0, 62 = 2, cl = 0 and dl = 4, with the total scores: 222 = 
7, 122-6,212 = 5, 112 = 4,221=3, 121 = 2, 211 = 1 and 111 = 0. 

In general, any particular ranking of profiles is determined by their pairwise rankings vis- 
i-vis each other. For an APS with x criteria and categories on each, and / profiles, a 
-v*(v'-l) 

maxmium of ^ pauwise rankings is possible. Thus for the exemplar APS with x 

= 3 and = 2, and eight profiles (i.e., 2^), there is a maximum of 28 [i.e., 8(8 - l)/2] 
pairwise rankings. 

The method of the mvention minimises the number of pairwise rankings that must be 
decided explicitly (via value judgements), such that, in this example, a minimum of two 
and a maxmium of four, rather than 28, is required, with the remaining pairwise rankings 
implicitly resolved as corollaries of the explicit rankings. The method of the invention 
comprises, in broad terms, three steps, as explained in tum below. The three main steps of 
the invration are illustrated in Figure 5, 

Step 1 of the method of the invention 510 involves identifying (or 'generating') the 
^ambiguities' of the APS that is being calibrated. 

Ambiguities are formed fi-om the total score equations of profile pairs whose pairwise 
rankings are a priori ambiguous, given the inherent inequaUties (as already explained, a2 
> al, 62 > 61 and c2 > cl in the present example). These are profile pairs in which one 
profile has a higher categorical rating on at least one criterion and a lower rating on at 
least one other criterion than the other profile. 

For example, the pairwise rankmg of profiles 221 and 212 — hereinafter referred to as 
"221 vs 212" and coixesponding to a2 + 62 + cl vs a2 + 61 + c2 — is ambiguous given 
a2 > al, 62 > 61 and c2 > cl, as noted above. 
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On the other hand, many profile pairs are unambiguously ranked. For example, 222 (a2 + 
b2 + c2) is always pairwise ranked first and 1 1 1 (al + 61+ cl) is always pairwise ranked 
second — e.g., a2 + b2 + c2> a\+bl-¥ c2 (122) and al+b2 + c2 (122) > al + fcl + cl, 
and so on. Other profile pairs are similarly unambiguously ranked, for example, a\ + b2 
5 +c2>al + M+c2(122>112),andsoon. 

As described above, identifying and eliminating all possible unambiguous pairwise 
rankings serves to cuU the rankings of the eight possible profiles fiom 8! permutations 
(40,320) to 48 (for illustrative purposes only, given strict preferences only and no ties) 
10 fi-om which the 12 in Figure 4 are further culled via another aspect of the internal logic of 
APSs described below. 

The method of the invention identifies and excludes profile pairs that are unambiguously 
ranked and then focuses exclusively on profile pairs that are ambiguously ranked. 



15 



Some ambiguously ranked profile pairs can be 'reduced' by cancelling variables that are 
common to both profile ecpiations. Thus a2 + 62 + cl a2 + 61 + c2 (221 vs 212, as 
above) can be reduced to 62 + cl vs 61 + c2 by cancelUng a2 from both profiles' 
equations. In effect, because a2 spears in both equations it has no bearing on the ranking 
20 of the two profiles that the equations represent. We refer to such reduced forms as 
'ambiguities'. 

Moreover 62 + cl vs 61 + c2 also corresponds to 121 vs 1 12, as al + 62 + cl vs al + 61 + 
c2 reduces to 62 + cl vs 61 + c2 after cancelUng the al teims from both profiles' 
25 equations. Thus 62 + cl 61 + c2 represents two ambiguously ranked profile pairs: 121 
vs 112 and 221 vs212. 

However, not all ambiguously ranked profile pairs are reducible in this fashion. For 
example, no variables can be cancelled firom a2 + 62 + cl vs al + 61 + c2 (221 vs 112), 
30 as none are common to both profiles' equations. Nonetheless, such irreducible 
ambiguously ranked profile pairs are also heremafler referred to as ambiguities. 
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Accordingly ambiguities can be classified by the number of criteria they contain, 
hereinafter referred to as the 'degree' of the ambiguity. Thus ♦2'^-degree' ambiguities 
contam two criteria, for example, bl + cl vj b\ + c2, as above, and *3"'-degree' 
5 ambiguities contain three criteria, for example, a2 + W + cl al + fcl + c2, as above, 
and so on. 

In general, the ambiguities for an APS with x criteria range from 2°'*-degree to x*-degree. 
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The algorithmically simplest process for generating ambiguities is to first create all / 
combinations of the y categories on the x criteria (that is, all profiles), and then pairwise 
rank them all against each other to identify pairs in which one profile has a higher 
categorical rating on at least one criterion and a lower rating on at least one otiier 

criterion. There will always be a total of pairwise rankings. 

As each ambiguously ranked profile pair is identified, it is reduced where possible by 
canceling variables in botii profiles' equations, and retained only if the resulting 
ambiguity has not ahieady been discovered. In otiier words, repUcated ambiguities are 
discarded. 



Accordingly tiie ambiguities for tiie exemplar APS with x = 3 and = 2 are set out in 
Figure 6, where tiie shaded ambiguities are duplicates of ones reported earlier in tiie table. 
In Figure 6, rankings tiiat are unambiguous are denoted by 'n.a.'. The blank elements of 
tiie matrix, except for tiie main diagonal, are mirror images of tiie ones reported, and 
25 shaded ambiguities are duplicates of ones rqxjrted earlier. 

Thus it can be seen in Figure 6 that although there are nine ambiguities in total, only six 
are unique: 



(\)b2+c\ vsbl + cl 

(2) a2 + cl vsa\+c2 

(3) fl2 + Mvjal+62 
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(4) a2 + b2 + cl vs al + bl + c2 

(5) a2 + M +c2vsal+b2 + cl 

(6) al+b2 + d2vsa2 + bl+ cl 



Clearly ambiguities (1) to (3) are 2"'*-degree ambiguities (and are duplicated in the figure) 
and ambiguities (4) to (6) are 3"'-degree ambiguities. 

In general, the number of ambiguities of each degree can be calculated fiom tiie two 
equations described below. 

For a giveai APS with x criteria and j; categories on each criterion, the total number of 
ambiguities (including replicates) of a givai degree (z) is given by: 



The first term corresponds to (the combinations formula); the second term is the 
powers of 2 and minus one; the third term is ^€2 raised to the ^"'-power or, alternatively, 
the sum of the first y - 1 natural numbers (e.g., 5(5-l)/2 =10=1+2 + 3 + 4) raised to 
the z*-power; and the last term is a power fimction. 

Of these ambiguities, the number of unique ambiguities (excluding rqplicates) of a given 
degree is given by: 



The two equations (1) and (2) differ only by the term (x) / " The meanmg of this and 
fh& other tturee terms (conunon to both equations) is discussed further below when an 
alternative process for generating ambiguities is described. 

Equations (1) and (2) can be compared with the equation mentioned earUer for the total 




(1) 




number of pairwise rankings (including unambiguous ones): 



2 
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Equation (2) is particularly useful because it reveals how many ambiguities at each 
degree must be resolved for any given APS. Figures 7 and 8 report the numbers of total 
and unique ambiguities for different APSs and degrees. As illustrated there, the number 
5 of ambiguities can be relatively large even for small values of x and y. For example, an 
APS with jc = 6 and j; = 4 (twice the values for the exemplar APS above) has a total of 
2,295,756 unique ambiguities across its five degrees (2°* to 6*). 

The simple process described above for generating ambiguities is further simplified if any 
10 profiles that are theoretically impossible are culled fi-om the set to be pairwise ranked 
before any ambiguities arc generated. 

For example, in the APS for hip or knee replacements referred to in Figures 1 and 2 it 
would be a contradiction for a patient to be rated as having "severe" "Pain on motion 
15 (e.g.. walking, bending)" while also being rated on another of the criteria as having the 
"Ability to walk without significant pain" for a distance of "over 5 blocks". Such a 
combination of categories on these two criteria is theoretically impossible and therefore 
all profiles that include it could be deleted fi^om the list to be pairwise ranked. 

20 Similarly, when validating an extant APS (rather than calibrating a new one), the profiles 
to be evaluated can be determined fi-om a 'stocktake' of the alternatives ranked by the 
APS over its lifetime. Any other profiles that might reaUstically be expected in the future 
could be added. 



This process also serves to increase the efficiaicy of Steps 2 and 3 of the method of the 
invention described below, as in general the more profiles that are culled, the fewer 
ambiguities there are, and therefore the simpler is the caUbration exercise. 

Notwithstanding such refinements, the simple process described above is computationally 
inefficient because profile comparisons that are ultimately unnecessary are performed and 
replicated ambiguities are discarded. 
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For example, to generate the above-mentioned 2,295,756 ambiguities for an APS wifhx = 
6 and = 4, as can be calculated from the data in Figure 8, 6,090,804 unnecessary profile 
comparisons are performed and 5,094,900 ambiguities are discarded. 

The particularly preferred process for generating ambiguities according to the invention is 
therefore described below. This is also the preferred process for the computer program of 
the invention. The main components of the process are illustrated in Figure 9. 

The particularly preferred process, hereinafter referred to as the 'efficient ambiguities 
generator', is described with explicit reference to the three terms in equation (2) above 
(reproduced below) and a specific example. The example is that of generating the 3^- 
degree ambiguities for an APS with x = 5 and j; = 3, of which there are 810 in total (as 
can be calculated from equation (2) and is reported in Figure 7). 

Although the equation applies to APSs with the same number of categories on the 
criteria, as explained later below, the process can be generalised to allow the number of 
categories to vary across criteria. For example, mstead of y = 3 for all 5 criteria in the 
above-mentioned APS, as is applied later, criterion a could have two categories, b three 
categories, c four, and so on. 



(x-z)!z! \ 7. ) 



(2) 



Equation {lys first term — ^ (= the combinations formula) — is the number 

of combinations of z criteria that can be selected from x criteria. 

For the above-mentioned APS with x = 5 (that is, criteria a, b, c, d and e\ 10 
combmations of three criteria (z = 3) are possible: abc, abd, abe, acd, ace, ade, bed, bee, 
bde and cde. Well-known algorithms exist for generating such combinations. 
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Each combination can be thought of as forming the 'base' for a group of ambiguities. For 
example, abc is the base for all S^'^-degree ambiguities centred on a + b + c vs a + b + c 
(hereinafter abbreviated to abc vs abc), such as a2 + M + c3 vs a3 + b2 + c2. 

5 

Because the criteria in each base (10 of them in the present example) have the same 
number of categories each, the bases can be ti^ated identically with respect to the 
following operations that correspond to the other two terms in equation (2). 

10 The second term ( 2'"' - 1 ) can be interpreted as representing tiie number of underlying 
•structiires' for a given degree (z). Structiares are generated by first listing the numbers 
between 1 and 2^* - 1 in binary form using z bits, where each bit represents a criterion of 
two categories r^resented by either 0 or 1 . 



Continuing witii tiie example of generating tiie 3"*-degree ambiguities for an APS witii x 
= 5, the structures for z = 3 are: 



Oil 
010 
001 



These structures conrespond to 3, 2, and 1 in binary fonn. For now, each structaare can be 
tiiought of as being analogous to a profile withjv = 2 for all criteria. Therefore only one 
ambiguity can be created firom each, which is done by finding each structure's 'Is 
complement', tiiat is, by 'flipping tiie bits'; thus in the example: 

2^ Oil vs 100 

010 vj 101 
001 vjllO 



Each term (eitiier a '0' or a *1') on eiflier side of tiie ambiguity structiire represents a 
criterion, and for each criterion tiieir relative magnitudes ('0* versus «1') represents tiie 
relative magnitixde of flie categories tiiey represent (i.e., 'low' or 'high'), such tiiat each 
structure has an imderlying pattern. 
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Thus on 100 in the example above (and analogously for 010 vs 101 and 001 vs 110) 

signifies that the first of the three criteria represented (corresponding to 0 vs I ) has 

a lower category on the left hand side (LHS) and a higher category on the right hand side 
(RHS) of each ambiguity that is derived firom it. 

Similarly, the second and third criteria (corre^onding to _1 1 vs J)0) both have higher 
categories on the UHS and lower categories on the RHS. 



The equation's third term 




is the number of ambiguities that can be 



generated firom each ambiguity stmcture, as determined by tiie number of categories (y) 
on the criteria in a given base. There are three steps to generating these ambiguities, as 
follows. 

First, all of the bases are matched with all of the structures. Li the present example, the 10 

bases {abc, abd, abe, acd, ace, ade, bed, bee, bde and cde) are matched with the three 

structures (01 1 100, 010 101 and 001 vsllQ) to produce 10 x 3 = 30 matches: 

abc vs abc with 01 1 vs 100 
abc vs abc with 010 vs 101 
abc vs abc with 001 1 10 
abd vs abd with 01 1 vs 100 
. . . and so on for another 26 matches. 

Second, the underlying ^pattern' for each match (that is, a base with a structure) is 
unplemCTited, according to the nimiber of categories on the bases. 

For abc vs abc with 01 1 vs 100, for example, with three categories on the criteria (y == 3), 

there are three ways each (that is, ^^"^^^^^ = 3(3 - l)/2 ways) of representing criterion a 

with a lower category on the LHS and a higher category on the RHS (0_ _ vs 1_ J, and 
criteria b and c both with higher categories on the LHS and lower categories on the RHS 
Lllvj^OO): 

a2 vs a3 b3 vs b2 c3 vs c2 
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al vs ah 
a\ vs al 



h2vsh\ 



c3 vs cl 
c2 vs cl 



Note that = ^€2, the number of combmations of categories (within the criterion) 

taken two-at-a-time. The ou^ut listed above embodies these combmations, where the two 
categories are simply ordered as reqviired by the structure. 





a2 + b3 + c3 vs a3 + b2 + c2 
a2 + b3 + c3 vs a3 + b2 + cl 
a2 + 63 + c2 vs a3 + b2 + cl 
a2 + b3 + c3 vs a3 + b\+ c2 
a2 + 63 + c3 vs a3 + 61 + cl 
a2 + 63 + c2 a3 + 61 + cl 
a2 + 62 + c3 vs a3 + 61 + c2 
a2 + 62 + c3 vs «3 + 61 + cl 
a2 + 62 + c2 vs a3 + 61 + cl 
al + 63 + c3 vs a3 + 62 + c2 
al + 63 + c3 vs a3 + 62 + cl 
al + 63 + c2 vs a3 + 62 + cl 
al + 63 + c3 V.S a3 + 61 + c2 
al + 63 + c3 vs a3 + 61 + cl 
al + 63 + c2 vs a3 + 61 + cl 
al + 62 + c3 vs a3 + 61 + c2 
al + 62 + c3 vs a3 + 61 + cl 
al + 62 + c2 vs a3 + 61 + cl 
al + 63 + c3 vs a2 + 62 + c2 
al + 63 + c3 vs a2 + 62 + cl 
al + 63 + c2 vs a2 + 62 + cl 
al + 63 + c3 vs a2 + 61 + c2 
al + 63 + c3 vs a2 + 61 + cl 
al + 63 + c2 vs a2 + 61 + cl 
al +624-C3 V5a2 + 61 +c2 
al + 62 + c3 vs a2 + 61 + cl 
al + 62 + c2 vs a2 + 61 + cl 
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Analogues of the three steps explained above are performed for all matches. For each of 
the 30 matches in the present example with jc = 5, = 3 and z = 3, 27 ambiguities 
analogous to the 27 above are generated, resulting in a total of 30 x 27 = 810 (unique) 3"*- 

degree ambiguities — i.e., — 4— x (2^-1-1) xfifc:^T= 10 x 3 x 27 = 810. 

(x—z)\zi \ 2 J 

5 

As noted earlier, flie process outlined above can be generaUsed to allow the number of 
categories to vary across the criteria. The key difference from the process explained 
above is that the underlying 'pattern' for each match (for example, abc vs abc with 01 1 
100) is idiosyncratic to the criteria included, as determined by the numbers of categories 
10 on each criterion. 

For example, if in the example refeired to above, instead of three categories on all 
criteria, criterion a has two categories, b has three and c four. Thus with just two 
categories for criterion a (for Oil vj 100, con^sponding to 0 vs 1 ), there is only one 

15 underiying pattern (ifcl> = 2(2 - 1)/2) corresponding to a lower category on the LHS 

and a higher category on the RHS of each ambiguity that is derived: 

a\ vs a2 

As before, with tiiree categories for criterion b (corresponding to _1_ vs _0_J, there are 
20 three underlying patterns (^fclH = 3(3 - l)/2) corresponding to a higher category on 

the IHS and a lower category on the RHS: 

63 vs bl 
M vs 61 
bl vs b\ 

Finally, with four categories for criterion c (corresponding to I vs ^0), there are sfat 

underlying patterns (ifcll = 4(4 _ i)/2) corresponding to a higher category on the 
LHS and a lower category on the RHS: 

c4 vs c3 



25 
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c4 vs c2 
c4 vs cl 
c3 vs cl 
c3 vs cl 
c2 vj cl 



By taking alll x 3 x 6 = 18 combinations of tiie above three sets of criteria-categories, all 
18 S^'^-degree ambiguities corresponding to ahc vs abc with Oil 100 (with two, three 
and four categcnies respectively) may be obtained: 



al 


+ W 


+ c4 


vs a2 + b2 


+ c3 


a\ 


+ b3 


+ c4 


vs a2 + b2 


+ c2 


al 


+ b3 


+ c4 


vs a2 + b2 


+ cl 


al 


+ b3 


+ c3 


vs a2 + b2 


+ c2 


al 


+ b3 


+ c3 


vs a2 + 62 


+ cl 


al 


+ b3 


+ c2 


V5 a2 + 62 


+ cl 


al 


+ b3 


+ c4 


vs a2 + 61 


+ c3 


al 


+ b3 


+ c4 


vsa2 + bl 


+ c2 


al 


+ b3 


+ c4 


vs a2 + 61 


+ cl 


al 


+ b3 


+ c3 


vs a2 + 61 


+ c2 


al 


+ b3 


+ c3 


vs a2 + 61 


+ cl 


al 


+ b3 


+ c2 


vs a2 + 61 


+ cl 


al 


+ 62 


+ c4 


vs a2 + 61 


+ c3 


al 


+ b2 


+ c4 


vs a2 + 61 


+ c2 


al 


+ b2 


+ c4 


vs a2 + 61 


+ cl 


al 


+ b2 


+ c3 


vja2 + 61 


+ c2 


al 


+ b2 


+ c3 


vs a2 + 61 


+ cl 


al 


+ b2 


+ c2 


vs a2 + 61 


+ cl 



25 



This process is performed for all matdies. But, because the numbers of categories for the 
30 criteria are different, each match generates an idiosyncratic set of ambiguities that 
depends on the numb^s of categories on the included criteria. 

For example, the nvumber of ambiguities generated from a6c vs abc with 01 1 vs 100 (as 
above) is different to the number from abd vs abd with 01 1 vs 100 when the numbers of 
35 categories on criteria c and are different. Therefore, in general the ambiguities for each 

(x-z)\z\ (combinations of z criteria from the x criteria) must be generated 
individually. 
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For example, in the case of the base corresponding to the particular combination of z 
criteria comprising the first z of the a: criteria {criterion 7, criterion 2, ... criterion z, 
where z < x), there are 2'"'-l structures as before. (The choice of the first z of the x 
criteria, instead of any other z of the x criteria, is for notational simpUcity.) Each structure 

I 2 j 2 J [ 2 J ^^^g^ties, where . . . j;^ are the 

numbors of categories an. each of these first z criteria. 

The total number of z*-degree ambiguities is obtained by summing the number of 

ambiguities (analogous to the example) across all — bases. In general, the 

(x-z)!z! 

10 number of ambiguities of each degree can be calculated from the equation described 
below. 



For a given APS with yu n, - yx categories on the respective x criteria, the equation is 
based on the following definitions. First. 7 is the set of the numbers of categories on the x 
15 criteria: 7= {yi, >>2, . . . y^} . Second, C is the set of unordered z-tuples formed by taking aU 

{x-zy.z\ of the elements of 7, z-at-a-time: C = {c | c is an 

unordered z-tuple from 7, as defined above}. Each of C's elements (i.e.. sets), c,, is 

numbered ftom 1 to — • c/ / = 1 1 Pa^i, ^ »„ i x 

{x-z)\z\ " ' ' ••• (x-z)\z\' ' yj' IS 

numbered from I to ziyy, J = 1,2, ... z. 

Applying these definitions, the number of unique ambiguities of a given degree (z), when 
the number of categories varies across criteria, is given by: 

(2-.-t)=<Tn^ (3) 



20 
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Alternatively this equation can be expressed as: (2'^"' -1) x SII'^^ "^^ • 

For common values ofyy (y,j=y), that is, aU criteria have the same numbers of categories, 
equation (3) is equivalent to equation (2) above. 

Either process explained above — for the same or, alternatively, different numbers of 
categories on the criteria — can be used to generate all of an APS's ambiguities of a 
given degree (such as S^'-degree, as above), or, alternatively, ambiguities can be 
generated individually. 



As each ambiguity is generated it is tested for whether or not it is theoretically 
impossible, and Iherefore to be discarded or not, given tiie theoretically impossible 
profiles that were culled earUer (as described above). For the simple process for 
generating ambiguities described earlier, all theoretically impossible profiles are simply 
1 5 removed before ambiguities are generated fi^om them. 

However, because the 'efficient ambiguities generator' explained immediately above 
does not generate ambiguities from profiles, the ambiguities must be tested as they are 
generated for whether or not the profile pairs that they represent have been culled or not. 

20 

For example, for an APS with x = 4 and = 3, the reduced form al + bS vs a3 + bl (i.e., 
13 — vs 32 ) represents nine ambiguously ranked profile paira: 

1311 vj 3211 

1312 3212 

2^ 1313 3213 

1321 VJ3221 

1322 vs 3222 

1323 vj 3223 

1331 3231 

1332 vs 3232 

1333 vs 3233 
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For al + b3 vs a3 + b2 to be theoretically impossible and therefore discardable, in all nine 
of these pairs at least one of the profiles — a minimum of nine and a maximum of 18 
profiles — must be theoretically impossible. If instead at least one of the nine pairs is not 
excluded, then al + 63 vs a3 + b2 is possible and therefore ought not to be discarded. 
5 Accordingly all nine pairs must be considered before it can be determined whether al + 
b3 vs a3 + bl should be discarded or not, but as soon as one profile pair is found that has 
not been excluded, the ambiguity should be retained. 

Enumerating all such profile pairs for any given ambiguity of degree z is relatively 
10 sti^^t-forward, as each profile pair is based on flie ambiguity in question, augmented by 
all possible combinations of the categories on tiie otiier x-z criteria. There are therefore 
such profile pairs when the number of categories on the criteria (y) is tiie same. 

As noted earlier, the term j/'"' appears in equation (1) above — giving tiie total number 
15 of ambiguities (including replicates) of a given degree — but not in the otiierwise 
identical equation (2) — giving tiie number of unique ambiguities (excluding replicates) 
of a given degree. 



Accordingly/-^ can be interpreted as the number of 'copies' of a particular ambiguity 
generated by tiie algorithmically simple process explained earlier. That is, in the example 
above witii X = 4 and j; = 3, each of tiie / - ^ = 3^ - 2 = 9 pairwise profile comparisons 
generates a\ + b3 vs a3 + bl (of which eight are discarded because they are replicates). 

Having generated tiie ambiguities. Step 2 of our metiiod involves explicitly resolving 
tiiem, one-at-a-time, while identifying all otiier ambiguities tiiat are implicitly resolved as 
corollaries. This step is represented in Figure 5 at 520. 



Any of tiie unresolved ambiguities of any degree could be selected for expUcit resolution 
during tiie caHbration process, however tiiere tend to be fewer ambiguities to store at 
lower degrees, and lower degree ambiguities are easier for decision makers to resolve, 
and so tiie process starts by resolving flie 2"^-degree ambiguities and tiien proceeds to 
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resolving successively higher-degree ambiguities. If instead it were desired that the 
number of decisions be minimised at the expense of the complexity of the decisions, the 
process should instead start by resolving the highest degree ambiguities and then proceed 
to resolving successively lower-degree ambiguities. 

5 

As ambiguities can only be rraolved via value judgements, they must be decided by an 
individual 'decision maker' or groiq) of 'decision makers', preferably with knowledge of 
the field in which the particular APS is to be ^lied. For example, a gcaap of decision 
makers for a medical APS may comprise a panel of doctors and patients. Hereinafter 
1 0 decision makers (plural) are referred to. 

Accordingly the preferences of the decision mak^s must be probed via a series of 
questions concerning their pairwise rankings of profiles. 

15 As listed earlier, the ambiguities for the original exemplar APS with x = 3 andy = 2 are: 
(1) b2 + cl vs bl + c2, (2) a2 + cl vs al + c2, (3) a2 + bl vs al + b2, (4) a2 + b2 + cl vs 
al + 61 + c2, (5) a2 + bl+ c2 vs al+b2 + cl and (6) al+b2 + c2 vs a2 + bl + cl. 

For ambiguity (1) b2 + cl vs bl + c2, for example, the decision makers are asked, in 
20 essence: "Given two alternatives that are the same witii respect to criterion a, which has 
the greater priority, _21 or _12?" 

If there is more than one decision maker, the process of getting answ«s to this and 
subsequent questions can be streamlined by asking decision makos to cast votes perhaps 
25 via email) for the pairwise rankings they favour. 

However, in this type of situation the majority voting runs the risks of the well-known 
voting paradox, whereby, depending on the decision makers' individual rankings of the 
profiles, tiie order the ambiguities are voted on can determine the resolutions that are 
30 derived. To avoid this possibility, were it likely, the decision makers should instead be 
required to reach a consensus on their pairwise rankings. 
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Logically, three mutually exclusive and exhaustive answers to the above question are 
possible; (1) _21 is strictly preferred to _12 or (2) _12 is strictly preferred to _21 or (3) 
they are equally preferred (i.e., indiffersaice between _21 and _12). 

5 

Notationally, these three preferences can be rq)resented as (1) _21 > _12 or (2) _12 > _21 
or (3) _21 = _12, corresponding to (1) b2 + cl>bl+ c2 or (2) bl+c2>b2 + cl or (3) 
b2 + cl=bl+ c2 (where, as usual, ">" is "strictly greater than" and *'=" is "equal to"). 

10 Weak preferaices, for ©cample, _21 is at least as preferred as _12 (notationally, _21 ^ 
_12), are also a logical possibility. However strict preferences are more useful from a 
practical perspective. 

The decision makers might, quite naturally, protest that fheir answer to the above 
15 question ("Given two alternatives that are the same with respect to criterion a, which has 
the greater priority, J21 or _12T') depends on whether criterion a is rated 'V or '2' (for 
both alternatives). Nonetheless, such distinctions are precluded by the mtemal logic of 
APSs. If the decision makers will not answer the question as it is posed then, in effect, the 
APS itself will answer it by default, since the ambiguity will eventually be implicitly 
20 resolved by the other explicitly resolved ambiguities chosen by the decision makers. 

More specifically, for example, if the decision makers were to decide that 121 > 112 — 
corresponding to &2 + cl > fel + c2 — then this implies 221 > 212, and vice versa. And 
analogously for M H-c2>fc2 + cl and61 +c2 = fe2 + cl. 

25 

The key word above is 'Hf\ as clearly neither inequality holds intrinsically. Therefore a 
value judgement is required to resolve ambiguity (1) b2 + c\ vs bl + c2: either 121 > 1 12 
and 221 > 212 or 1 12 > 121 and 212 > 221 or 121 ^112 and 221 = 212. By virtue of the 
laws of arithmetic, if one inequality or equality holds then the other must too; if one does 
30 not then neither does the other. It is impossible to have one half of either proposition 
without the other. 
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Accordingly the question above could be rephrased, in essence, as: "Which one of the 
following three possible rankings of two alternatives do you prefer, (1) 121 > 112 and 
221 > 212 or (2) 112 > 121 and2\2 > 221 or (3) 121 = 112 a/irf221 =212?" 

Continuing with the example, suppose ttiat in fact flie decision makers resolve ambiguity 
(1) by choosing _21 > _12 (in other words 121 > 112 am/ 221 > 212), corresponding to 
bl + cl > b\ + c2. Two alternative, but equivalent, approaches are available for 
identifying the implicitly resolved ambiguities. 

Although these approaches — hereinafter referred to as 'Approach 1' (of which there are 
two variants) and 'Approach 2' — differ in the means and the sequence in which the 
impUcitiy resolved ambiguities are identified, both generate the same list of expUcitiy 
resolved ambiguities, from which the point values are derived at Step 3 described below. 

However because Approach 1 becomes relatively unwieldy, and therefore more resource 
intensive to implement, for more criteria and categories than the exemplar APS with ;c = 3 
and j; = 2, the particularly preferred embodiment of the computer program of the 
invention is based on Approach 2. Nonetheless, Approach 1 is described first as it is more 
intuitively tractable and is therefore usefiil for illusti-ating ^proach 2. 

Approach 1 may be summarised as follows. After a given ambiguity is explicitly resolved 
by tiie decision makers, all other ambiguities tiiat are impUcitiy resolved as corollaries are 
immediately identified by adding appropriate inherent inequalities and/or oflier expUcitiy 
resolved ambiguities (inequalities or equalities). Then another (unresolved) ambiguity is 
expUcitiy resolved by the decision makers and all its corollaries are identified. The 
process is repeated until all ambiguities have been resolved, eiflier expUcitiy or implicitly. 

Thus a coroUary of tiie e:q>licit resolution of ambiguity (1) as 62 + cl > 61 + c2 (decided 
by tiie decision makers, as explained earUer) is tiie implicit resolution of ambiguity (4) al 
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+ Z?2 + cl V5 al + Z?l + c2. This is revealed by adding the inherent inequality a2 > al and 
b2^cl>b\ +c2: (a2>al) + (Z>2 + cl>M + c2) = (a2 + 62+cl>al + il+c2). 

Although this addition is mathematically legitimate, its Iheoretical validity in the context 
5 of APSs rests on the assumption that the decision makers are logically consistent in the 
sense that their pairwise profile rankings are transitive. Transitivity means in general that 
if alternatives^ > B mdB> C, then>4 > C. 

Thus, in the present example, the explicit resolution of ambiguity (1) as W + cl > 61 + c2 
10 corresponds to 121 > 112 and 221 > 212, as discussed earlier. Moreover 212 > 112 
because a2 > al. Therefore, assuming profile rankings are transitive, 221 > 212 and 212 

> 112 implies 221 > 112 — correspondmg to a2 + W + cl > al + 61 + c2, as was 
revealed above by adding a2 > a 1 and 62 + c 1 > 1 + c2. 

15 As there are no other corollaries at this point, the next ambiguity on the list, in this case 
ambiguity (2) a2 + cl vs al + c2, can be presented to the decision makers to resolve via 
an analogoxis question to the first one: "Given two alternatives that are the same with 
respect to criterion 6, which has the greater priority, 1_2 or 2_1?" 

20 Suppose the decision makers answer 1_2 > 2_1, corresponding to al + c2 > a2 + cL A 
corollary is the implicit resolution of ambiguity (6) al + 62 + c2 a2 + 61 + cl, as 
revealed by adding inherent inequality 62 > 61 to al + c2 > a2 + cl: (62 > 61) + (al + c2 

> a2 + cl) = (al + 62 + c2 > a2 + 61 + cl). 

25 Li addition, both (1) 62 + cl > 61 + c2 and (2) al + c2 > a2 + cl implicitly resolve 
ambiguity (3) a2 + 61 vs al + 62, as revealed by their addition: (62 + cl > 61 + c2) + (al 
+ c2 > a2 + cl) = (al + 62 > a2 + 61) (after cancelling the c terais). 

As for the addition of inherait inequalities and explicitiy resolved ambiguities (as above), 
30 this addition is also justified by the assumption that the pairwise profile rankings of the 
decision makers are transitive. 
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Thus, in the present example, the aq>licit resolution of ambiguity (1) as i>2 + cl > 61 + c2 
corresponds to 121 > 1 12 and 221 > 212, and (2) al + c2 > a2 + cl corresponds to 112 > 
211 and 122 > 221. Given 121 > 112 and 112 > 211, then, by transitivity. 121 > 211 is 
implied. Likewise, given 122 > 221 and 221 > 212, then 122 > 212 is impUed. Both 121 
> 21 1 and 122 > 212 correspond to al + &2 > ^2 + fcl, as was revealed above by adding 
(1) 62 + cl > £»1 + c2 and (2) al + c2 > a2 + cl. 

Thus fixjm just two expUcit decisions to resolve ambiguities (1) and (2), another three 
ambiguities (3, 4 and 5) are impUcitly resolved, so that five of the six ambiguities are 
resolved. 



The remaining ambiguity, ambiguity (5) a2 + 61 + c2 vs al + 62 + cl, must be explicitiy 
resolved by the decision makers, via a question that is conceptuaUy simpler than the two 
15 earUer ones: "Which alternative has flie greater priority, 212 or 121?" Suppose tiie 
decision makers answer 212 > 121, corresponding to a2 + 61 + c2 > al + 62 + cl. 

The system is now fully specified as: (1) 62 + cl > 61 + c2, (2) alH- c2 > a2 + cl. (3) al 

+ 62>a2 + 61,(4)a2 + 62 + cl>al+61+c2,(5)a2 + 61+c2>al+62 + cland(6) 
20 aH- 62 + c2 > a2 + 61 + cl — as well as the inherent inequalities al > a\, bl > 61 and 
c2 > cl . Of inequalities (1) to (6), only tiiree (1, 2 and 5) were expUcitiy resolved by tiie 
decision makers, with tiie other tiu-ee (3, 4 and 6) implicitly resolved as corollaries. 



25 



30 



Finally, in general but not in tiie present example, any expUcitiy resolved ambiguities that 
are tiiemselves corollaries of otiier explicitiy resolved ambiguities can be removed from 
ttie list from which tiie point values are derived at Step 3 (below). This is because only 
independent inequaUties/equalities are required for deriving point values. 

A variant of tiie approach outiined above — in effect, its converse — is ttie identification 
of tiie sufficient (but not necessary) conditions for implicitiy resolving ambiguities, in 
terms of (otiier) resolved ambiguities. Ambiguities are eitiier expUcitiy resolved by tiie 
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decision makers or impUcitly resolved when their sufficient conditions are met. Any 
ambiguities whose sufficient conditions are not met must therefore be resolved explicitly 
by the decision makers, until all ambiguities have been resolved, either expUcitly or 
impUcitly. 

Thus in the exemplar APS with x = 3 and = 2, ambiguities (4). (5) and (6) have four 
sufficient conditions each in terms of resolved 2'^-degree ambiguities. Specifically, 
ambiguity (4) is (impUcitly) resolved as a2 + 62 + cl > al + fel + c2 if at least one of the 
foUowing inequalities holds: o2 + cl > «1 + c2 or a2 + cl = al + c2 (in both cases 

because 62 > M) or 62 + cl > M + c2 or 62 + cl = 61 + c2 (in both cases because fl2 > 
al). 



These and analogous sufficient (but not necessary) conditions for ambiguities (5) and (6) 
to be impUcitly resolved as a2 + 61 + c2 > al + 62 + cl and al + 62 + c2 > a2 + 61 + cl 
15 respectively are Usted in Figure 10. No sufficient conditions exist for the opposite 
resolutions of the three ambiguities nor for equalities (that is, not for RHS > LHS nor 
LHS = RHS) in terms of resolved 2'"'-degree ambiguities. 

It is then simply a matter of comparing these conditions against resolved ambiguities (1), 
20 (2) and (3) (arrived at via Approach 1 explained above). Accordingly, as identified via 
shading in Figure 10, (1) 62 + cl > 61 + c2 impUcitly resolves ambiguity (4) as a2 + 62 + 
cl > al + 61 + c2, and both (2) al + c2 > a2 + cl and (3) al + 62 > a2 -f- 61 implicitiy 
resolve ambiguity (6) as al + 62 + c2 > a2 + 61 + cl. 

25 These are the same S'^-degree resolutions as were revealed earUer via the expUcit 
resolutions of ambiguities (1) and (2) and their additions to inherent inequaUties and each 
other respectively. As such they reflect, as before, pauwise profile rankings that are 
transitive. 

30 Finally, as before, ambiguity (5) remains to be e^Ucitly resolved, and then the system is 
fully specified. 
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This variant of Approach 1 geueralises for APSs with higher-degree ambiguities. Their 
sufficient conditions are in terms of both individual lower-degree resolved ambiguities 
and combinatiotis of them, of which there may be many. For example, the sufficient 
5 conditions for the resolution of a 5^-degree ambiguity are m terms of combinations of 
l^'^-degree resolved ambiguities, 3*^-degree resolved ambiguities and 4*^-degree resolved 
ambiguities. 

However, sufficient conditions for an ambiguity of a given degree can also be identified 
10 in terms of resolved ambiguities of the same, or even higiher, degrees. In the present 
example, sufficient conditions can also be identified for resolving 2''*^-degree ambigtiities 
in terms of other resolved 2°*^-degree ambiguities. For example, a sufficient condition for 
resolving ambiguity (3) a2 + bl vs al + b2 as al + 62 > a2 + bl is (1) Z?2 + cl > fel + c2 
and (2) al + c2 > a2 + cl, corresponding to (62 + cl > 61 + c2) + (al + c2 > a2 + cl), as 
1 5 explained earlier. 

This means that in general it is difficult to enumerate and check all possible sufficient 
conditions (to ensure that none are missed); therefore in practice, this variant must be 
supplemented by other methods, such as the first variant of Approach 1 explained above. 

20 

Finally, with respect to both variants of Approach 1, because ambiguities (1) to (3) are 
not independent, both the order and the maimer in which they are resolved affects the 
number of explicit value judgements that are reqiiired. The maximomi number required is 
four and the miniTrnm is two. 

25 

For example, if inequality (3) had been decided before mequalities (1) and (2) (instead of 
after, as above), then all three ambiguities (as well as ambiguity 5) would have had to 
have been resolved explicitly. This is because inequality (3) al + 62 > a2 + 61 cannot be 
added to (1) 62 + cl > 61 + c2 or (2) al + c2 > a2 + cl to obtain the other inequality, and 
30 yet inequalities (1) and (2) imply (3), For ambiguities (4) to (6), on the other hand, one 
and only one must be resolved explicitly. 
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Alternatively, for example, had ambiguity (1) been resolved as b2 + cl = bl + c2 instead 
of b2 + cl > bl + c2 (indifference rather than strict preference), and ambiguity (2) 
resolved as aH- c2 > a2 + cl (as above), then no other explicit resolutions would have 
5 been necessary. 

This can be confirmed by noting that (1) b2 + cl=bl+ c2 implies both (4) a2 + b2 + cl 
>aH-M +c2 and (5) a2 + M + c2>al +62 + cl. Mother words (fc2 + cl=M +c2) + 
(62 > bl) for both of them and (2) al + c2 > a2 + cl impUes (6) al + 62 + c2 > a2 + 61 + 
10 cl (as above) and (62 + cl = 61 + c2) + (al + c2 > a2 + cl) impUes (3) al + 62 > a2 + 
61. Qearly, the point values daived horn this system of equations and ineqtialities would 
be different to the point values derived earlier. 

Similarly the system would be completely specified by the explicit resolution of 
15 ambiguities (1) and (6) as 61 + c2 > 62 + cl and a2 + 61 + cl > al + 62 + c2 only. Of 
particular interest is: (a2 + 61 + cl > al + 62 + c2) + (62 > 61) = (2) (a2 + cl > al + c2); 
and (a2 + 61 + cl > al + 62 + c2) + (c2 > cl) = (3) (a2 + 61 > al + 62). This illustrates 
the fact that it is not necessary to resolve lower-degree (here 2°''-degree) ambiguities 
before implicitly resolving higher- degree (here 3"*-degree) ambiguities. The process can 
20 be reversed, as illustrated here. 

Unfortunately, as mentioned earlier, both variants of Approach 1 become unwieldy for 
APSs with more criteria and categories than the exemplar APS with jc = 3 and;; = 2. This 
is because calculating and managing all possible combinations of additions or possible 
25 sufficient conditions is resource inten^ve. 

The 'additions' variant of Approach 1 involves maintaining a list of all possible 
inequalities/equalities that result &om the addition of each e^qplicitly resolved ambiguity 
witii each other explicitly resolved ambiguity as well as with the inherent inequalities, 
30 and with every sum generated, and each of these sums with each other recursively until 
all possible additive combinations are exhausted. Each newly generated ambiguity is 
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'checked off against the list of implicitly resolved ambiguities: if it is not on the list then 
it is yet to be resolved. 

Similarly, the 'sufficient conditions' variant of Approach 1 involves managing a list of all 
5 implicitly resolved ambiguities, identifying all possible sufficient conditions of lower 
degrees, of which there may be a great many combinations, and identifying ambiguities 
implied by Ihe same or higher degree explicitly resolved ambiguities (by some other 
means). 

10 Approach 2, on the other hand, which is explained below, is more efficient and is 
therefore the preferred method used in the computer program of the invention. 

In smnmary, Approach 2 involves testing ambiguities individually for whether or not they 
are implicitly resolved as corollaries of the explicitly resolved ambiguities (that were 
15 resolved earlier). If a given ambiguity is identified as having been implicitly resolved 
then it is deleted. If instead it is not implicitly resolved then it must be explicitly resolved 
by the decision makers. The process is rqpeated until all ambiguities have been identified 
as having been implicitly resolved or they are explicitly resolved. 

20 Thus, with reference to the exemplar APS with x=3 and y =2, after the decision makers 
resolve ambiguity (1) as 62 + cl > felH- c2 as described earlier, the next ambiguity on the 
list ((2) al + c2 vs a2 + cl) is tested for whether or not it is implicitly resolved as a 
corollary of 62 + cl > 61 + c2, as well as the inherent inequalities a2 > al, 62 > 61 and 
c2>cl. 

25 

Here, as in the computer program of the invention, this and subsequent tests can be 
performed via linear programming. In effect, this test is performed by asking the 
following two hypothetical questions (of the method, not the decision makers). 

30 Hypothetical Question 1: If it were the case that the ambiguity in question [here (2) al + 
c2 vs al + cl] had been implicitly resolved as LHS > RHS (i.e., al + c2 > a2 + cl), then 
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does a solution exist to the system comprising this hypothetical mequality and the (actual) 
explicitly resolved inequalities/equalities [here (1) b2 + cl > bl + c2] — as well as the 
inherent inequalities [here a2 > al, bl > bl and c2 > cl]? (Yes or iVb?) 

iTthe answer is No — and therefore it would not be theoretically possible for the decision 
makers, if they wanted to, to explicitty resolve the ambiguity in question as LHS > RHS 
(here al + c2 > a2 + cl) — then it must be true that either RHS > LHS or LHS = RHS 
(i.e., either a2 + cl > al + c2 or al + c2 = a2 + cl). This impUes that the ambiguity in 
question (here ambiguity 2) has been implicitly resolved (i.e., as either a2 + cl > al + c2 
or al + c2 = a2 + cl). Hence it is of no further use and can be deleted. 

^instead the answer to Question I is Yes — and therefore it would be theoretically 
possible for the decision makers, if they wanted to. to expUcitly decide LHS > RHS (here 
al + c2 > a2 + cl) — then the followmg second hypothetical question is asked. 

Hypothetical Question 2: If it were instead the case that the ambiguity in question (here 
ambiguity 2) had been impUcitly resolved as RHS > LHS (that is, a2 + cl > al + c2), 
then does a solution exist to the system comprising this hypothetical inequality and the 
(actual) expUcitly resolved inequaUties/equalities [here (1) b2 + cl>bl+ c2, as before] 
— as well as the inherent inequalities [here a2 > al, > 61 and c2 > cl]? (Yes or No?) 

If±e answer is No — and therefore it would not be theoretically possible for the decision 
makers, if they wanted to, to explicitly resolve the ambiguity in question as RHS > LHS 
(here a2 + cl > al + c2) — then it must be true that either LHS > RHS or LHS = RHS 
(either al + c2 > a2 + cl or al + c2 = a2 + cl). This implies that the ambiguity in 
question (here ambiguity 2) has been implicitly resolve4 in this case as al + c2 > a2 + 
cl or al + c2 = a2 + cl. Hsaice it is of no further use and can be discarded. 

i^instead the answer to Question 2 is Yes then it must be infenred that as well as it being 
30 theoreticaUy possible for the decision makers, if they wanted to, to expUcitly resolve the 
ambiguity in question as DHS > RHS (here al + c2 > a2 + cl) from Question 1 that it is 
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also theoretically possible for them to resolve it as RHS > LHS (a2 + cl > al + c2). This 
implies that the ambiguity in question (here ambiguity 2) is not implicitly resolved as a 
corollary of the explicitly resolved ambiguities, and therefore it must be explicitly 
resolved by the decision makers. 

5 

In the case of ambiguity (2) a\ + c2 vs al + c\ the answers to Questions 1 and 2 are Yes 
and Yes, and so the ambiguity should be presented to the decision makers for them to 
explicitly resolve. As for the earlier demonstration of Approach 1, suppose it is decided 
al+c2>fl2 + cL 

10 

The next ambiguity on the list — (3) al + b\ vs al + b2 — is then tested via the same 
process as outlined above. This time, though, the list of (actual) explicitly resolved 
inequalities/equalities against which (3) a2 + &1 > al + b2 (i.e., LHS > RHS, as for 
Question 1) and then, if necessary, (3) al-hb2>a2 + bl (RHS > LHS, as for Question 2) 
15 are tested comprises (2) aH- c2 > a2 + cl as well as (1) 62 + cl > fel + c2 (as before). 

Thus the list of explicitly resolved inequalities/equalities is continually updated — 
including in general but not in the present example, as for Approach 1, the identification 
of any explicitly resolved inequalities/equalities that are themselves corollaries of others 
20 on the list. 

The process is repeated for all ambiguities until all of them have been identified as having 
been implicitly resolved or they are explicitly resolved by the decision makers. This can 
be sunmaarised for the present example as follows. 

25 

For ambiguity (3) the answer to Question 1 is No, and therefore this ambiguity is 
identified as having been implicitly resolved as a corollary of the explicitly resolved 
ambiguities. For ambiguity (4), the answer to C^estion 1 is Yes but the answer to 
Question 2 is No, and therefore this ambiguity is identified as having been implicitly 
30 resolved. 
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For ambiguity (5) the answers to both questions are Yes, implying that this ambiguity has 
not been implicitly resolved, and so it must be explicitly resolved by the decision makers: 
as a2 + 61 + c2 > al + 62 + cl, as for the demonstration of Approach 1. Finally, for 
ambiguity (6) the answers are Yes and No, and therefore this ambiguity is also identified 
5 as having been implicitly resolved. 

Thus, of the six ambiguities, three had to be explicitly resolved by the decision makers — 
(1) 62 + cl > 61 + c2, (2) al + c2 > a2 + c\ and (5) a2 + 61 + c2 > al + 62 + cl (the 
same three as for Approach 1) — with the other three (3, 4 and 6) identified as having 
10 been implicitly resolved as corollaries. 

The final step of the method (Step 3) of the invention involves simultaneously solving the 
system of (independent) explicitly resolved ambiguities (inequalities and equalities) and 
inherent inequalities to obtain the point values. This step of the method is represented in 
15 Figure 5 at 530. 

As described above, any explicitly resolved ambiguities that are themselves corollaries of 
other explicitly decided inequalities are removed, because only independent 
inequalities/equaUties are required for deriving the point values. 

20 

For the exemplar APS with x = 3 and 3; = 2, one solution for (1) 62 + cl > 61 + c2, (2) al 
+ c2 > a2 + cl and (5) a2 + 61 + c2 > al + 62 + c\ and a2 > al, 62 > 61 and c2 > cl is: 
a\ = 0, a2 = 2, 61 = 0, 62 = 4, cl = 0 and c2 = 3. These point values produce ranking #9 
in Figure 4. 

25 

The method explained above may be sununarised as follows. First (Step 1), generate the 
ambiguities of the APS that is to be calibrated as shown at 5 10 in Figure 5. 

Then (Step 2) ejq^licitly resolve them via the value judgements of the consulted decision 
30 makers, while identifying all other ambiguities that are implicitly resolved as corollaries, 
until all ambiguities are resolved. This step is illustrated at 520 in Figure 5. Although, in 
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theory, the implicitly resolved ambiguities can be identified vi^ Approach 1 or Approach 
2 (both explained above), the latter is more efficient and therefore it is the particvilarly 
preferred method for the computer program of the invention, as explained below. 

5 Finally (Step 3), simultaneously solve the system of explicitly resolved ambiguities (that 
is, inequalities and equalities) and inherent inequalities to obtain tiie point values for the 
APS. This step is illustrated at 530 in Figure 5. 

The property (assumption) that the profile rankings of the decision makers are transitive 
10 (or logically consistent), enables the number of ambiguities that must be explicitly 
resolved at Step 2 to be nmmnised, with the remainder emerging implicitly as corollaries. 

For APSs with more criteria and categories than the exemplar with x = 3 and y ^ 2, 
minimising the number of explicitly resolved ambiguities, and accordingly the number of 
15 value judgements required firom the decision makers, is a significant practical advantage 
of our method. 

The computer program for implementing the method described above will now be 
described more particularly. The computer program of the invention comprises, in broad 
20 terms, five steps, as explained in turn below. An overview of the five steps is illustrated 
in Figure 11. 

Before starting the program, the user — who may be one of the decision makers 
consulted to resolve the ambiguities, or, alternatively, he or she may be a facilitator of the 
25 calibration process — must have chosen the criteria and categories of the APS that is to 
be calibrated and ranked each criterion's categories. 

At Step 1, when the program begins, the user may be asked to enter a title for the APS 
and the criteria and their categories, and to rank Ihe categories for each criterion. The 
30 criteria and categories must be labelled in terms of the variables (e.g. 'al\ ^aV *a3\ etc.) 
and verbally described in preparation for their later presentation to the decision makers. 
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The user is givai the opportunity of listing theoretically impossible combinations of 
criteria and categories that partially or fully specify profiles — known as the To Be- 
Excluded list — and that are therefore to be used to cull ambiguities that are generated by 
5 the program. This list may be left empty if the user wishes. 

After the program is initialised, it calculates the number of unique ambiguities to be 
resolved using equation (3) above. This can be reported to the user and used to estimate 
whether the system can be solved in an acceptable amount of time giveu the computing 
10 resources that are available. 

Step 2 of the program involves generating the ambiguities usmg tiie 'efficient ambiguities 
goierator' described earlier. An overview of this step is illustrated in Figure 12. 

15 For simplicity and efiBciency, ambiguities are generated one degree at a time, beginning" 
with the 2° -degree, as determined by the value of a control variable — known as the 
Current Degree variable — which is initially set to 2. 

As each ambiguity is generated it is checked for whether or not it is theoretically 
20 possible, given the To Be Excluded list of partially or fully specified profiles. If the 
ambiguity is theoretically possible it is then tested via ^proach 2 of Step 2 of the 
method of the invention explained earlier for whether or not it is implicitly resolved by 
the explicitly resolved ambiguities. An overview of the procedure is illustrated in Figure 
13. 

25 

This and similar tests may be paformed via linear programming, which is described 
below. 

If an ambiguity is found to be implicitly resolved then it is discarded; otherwise it is 
30 added to a list known as the To Be Resolved list. (Note that when the 2™*-degree 
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ambiguities are generated, none will be implicitly resolved because none have yet been 
explicitly resolved.) 

Step 3 of the program is to present the To Be Resolved Ust of unresolved ambiguities for 
the current degree to the user. An overview of this step and Stsp 4 (explained below) is 
illustrated in Figure 14. 

The user can choose to view the ambiguities either m equation form (for example, a\ + 

b2 vs a4 + 61) or symboUcally (for example, 21 or 41 ), which are ordered 

either randomly or by their criteria and categories. The user is invited to select an 
ambiguity for the purpose of explicitly resolving it. The selected ambiguity is 'translated' 
verbally in terms of flie criteria and category descriptions. 

If the user desues. he or she may skip this particular ambiguity and select another one, or 
15 she may resolve it by clicking one of three buttons labelled (m essence): "LHS greater" or 
«RHS greater" (i.e., < or RHS preferred) or "LHS and RHS equal". If "RHS greater 
(preferred)" is chosen, for consistency, the LHS and RHS of the resolved ambiguity are 
switched and stored as RHS > LHS. 



The system can also permit weak inequalities (such as "LHS greater than or equal to" and 
"RHS greater than or equal to"), however this means that some ambiguities will later be 
partially (weakly) solved and so some buttons in the user mterface must be disabled when 
the ambiguity is selected. Note that in any resulting APS the result will be either "greater 
than" or it will be "equal to" but it will not be both. The system can also be designed so 
that only strong inequalities and no equalities are permitted, thereby producing strict 
profile rankings only. 



Step 4 is for the program to remove the expUcitly resolved ambiguity (as above) from the 
To Be Resolved list and add it — as either an inequality or equality (depending on how 
30 the ambiguity was resolved) — to the Ust of explicitly resolved ambiguities, known as the 
Ejq?licitly Resolved list. An overview of this step and Step 3 is illustrated as Figure 14. 



60669-1 



40 

Each inequality/equality on the Explicitly Resolved list is then tested as to whether or not 
it is implied by the others on the list. Any found to be implied may be marked as being 
'redundant* and hereinafter ignored, but not deleted because they may be re-used later if 
5 any explicitly resolved ambiguities are later revised by the decision makers (explained 
below). 

All ambiguities on the To Be Resolved list are then tested for whettier or not their 
resolution is implied by the (non-redundant) inequalities/equalities on the Explicitly 
10 Resolved list, and if so they are deleted from the To Be Resolved list. This test was 
explained in earlier in terms of Hypothetical Questions 1 and 2 of Approach 2 of the 
method's Step 2 and is illustrated in Figure 13. 

In essence, the test involves finding whether or not a solution (in terms of feasible point 
15 values) exists to a system comprising the explicitly resolved inequalities/equalities and 
inherent inequalities, and each of the possible hypothetical inequalities in tum (that is 
LHS > RHS and RHS > LHS) corresponding to the ambiguity in question. This and the 
earlier tests based on determining the existence of solutions are performed via linear 
programming (LP) with inequality and equality constraints. LP is discussed in more detail 
20 in the final section below. 

If the To Be Resolved list is not empty, the program returns to Step 3. If instead the To Be 
Resolved list is empty and the Current Degree is equal to the number of criteria (x) in the 
particular APS being calibrated, the program proceeds to Step 5, Alternatively, if the 
25 Current Degree is less than the APS's mmiber of criteria (and the To Be Resolved list is 
empty), the Current Degree is increased by * T and the program returns to Step 2. 

The user may choose at any time to view flie explicitly resolved ambiguities for all 
degrees and *undo* any of them. Explicitly resolved ambiguities that are 'redundant* are 
30 displayed in a different colour to al^ the user to the fact that undoing them alone will 
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have no material effect An overview of the 'undo module', as referred to in Figure 1 1, is 
illustrated in Figure 15. 

When expUcitly resolved ambiguities are undone, they are deleted fiom the Explicitly 
Resolved list. If any non-redundant explicitly resolved ambiguities are undone, the 
explicitly resolved ambiguities marked redundant are re-tested for whether or not they are 
still redundant, and marked accordingly. 

Also, if non-redundant ambiguities are undone, the Current Degree variable is reset to the 
degree of the lowest-degree ambiguity that was undone, and the program restarts from the 
second step in the computer program as set out above. 

When all the ambiguities for all degrees have been resolved, tiie list of expUcifly resolved 
inequaUties/equaUties and inherent inequaUties is solved via linear programming for tiie 
point values of the APS. These are presented to useis, as well as a range of 'summary 
statistics', including tiie numbers of ambiguities and expUcitiy resolved ambiguities at 
each degree and, it tiie user wants tiiem, the inequaUties/equalities that were chosen. 

In tiic interests of deriving point values tiiat are integers and as low as possible, thereby 
maximising tiieir 'user-friendlmess', integer programming and an objective function that 
minimises tiie sum of tiie variables (for example, a\+ a2 + ai + b\+b2 + b3 + c\+c2 + 
c3, etc.) maybe specified. 

Alternatively, were it desired by users tiiat tiie point values be normalised to a particular 
range for tiie profile total scores, such as 0 to 100. tiie derived point values are scaled 
accordingly. However tiiis usually forces some point values to be fiactions. which is not 
as 'user-fiiendiy as integers. Accordingly botii integer and normaUsed values can be 
calculated and presented. 

As explained above, linear programming (LP) may be used in tiie computer program of 
the invention. Specifically. LP may be used at Steps 2 and 4 to determine whettier flie 
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resolution of an ambiguity is implied by the inequalities/equalities on the Explicitly 
Resolved list. In Step 5, LP is used for deriving the point values when all ambiguities 
have been resolved. The following features must be observed when LP is used. 

5 First, the inequalities/equalities and inherent inequalities must be converted to a form 
suitable for LP. For example, the inherent inequalities di>a2> al must be written as a2 
- al = 1 and fl3 - a2 = 1, and the explicitty resolved ambiguities a2 + M = a3 + bl and 
a2 + 63>a3+fc2asa2 + i»2-a3-M = 0anda2 + fc3-fl3-fe2=l. Setting the RHS of 
the weak inequaUty to "1" (that is, 'q)silon') corresponds to the initial inequaKty being 
10 strict, although other values for epsilon may perform at least as well. 

Second, m Steps 2 and 4, the LP objective fimction need only be "0", as all that is bemg 
tested for is the existence of a solution rather than a particular optimal solution. 

15 Finally, because the variables corresponding to the lowest category on the respective 
criteria (al, bl, cl, and so on) are, effectively, 'numeraires' or 'baseline' values for the 
respective criteria, they can be set equal to zero, thereby eliminating them from the 
system to be solved and increasing the efficiency of the LP algorithm. 

20 Below is an illustration of how LP may be used to test whether or not an ambiguity on the 
To Be Resolved list is implied by the inequalities/equalities on the Explicitly Resolved list 

For the sake of the example, suppose the ambiguity m question isal+b3 + c3vsa2 + bl 
+ cl and the following inequalities/equalities are on the Explicitly Resolved list (where 
25 the shaded uiequality signifies that it is 'redundant' in the sense, as discussed earlier, that 
it is implied by others on the list). 

a2 + fc2 = a3+61 
a2 + 63>a3 + 62 

30 a2 + cl>al+c3 

61+c2 = M+cl 
M +c3 = M +c2 
al+62 + c3>a2 + M+cl 
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In addition, the inherent inequalities of this exemplar APS with x = 3 and;); = 3 are: 

a3 > a2 > al 
b3>b2>bl 

^ c3 > c2 > cl 

These inequalities/equaUties (and ignoring the shaded redundant inequality) — and given 

al = £>1 = cl = 0 (as discussed above) — are represented in a form suitable for LP, as 

foUows. By definition, a solution (in terms of the point values) exists to the LP pioblan: 

minimise 0 
subject to: a2 + b2-a3 = 0 

a2 + b3-a3-b2^l 
a2-ci^\ 
c2-b3 = 0 
c3-b3-c2 = 0 
b2 + c3-a2^1 
a2^1 
a3-a2^l 
b2^1 
b3-b2^l 
c2>\ 
c3-c2^ 1 



15 



20 



To test whether a\^^b3 + c3 vs a2 + bl+ cl is implied by the inequalities/equaUties on 
25 the Explicitly Resolved list, the above LP problem is first augmented with W + c3 - a2 ^ 
1 (i.e., al+b3 + c3>a2 + bl + cl), and tested for whether or not a solution to this new 
problem exists. 



30 



A solution does exist, and so the original LP problem is next augmented with a2-b3- 
c3 ^ 1 (i.e., a2 + bl+cl>al+b3+ c3) and tested for whether or not a solution to this 
second new problem exists. 



In this case there is no solution, and so it must be inferred that either al + M + c3 > a2 + 
bl + cl or al+b3 + c3^ca + bl+ cl is implied by the inequaUties/equalities on the 
35 Explicitly Resolved list. The first inequality above is. easily confirmed here by adding b3 
> b2 to the last expUcitly resolved ambiguity on the first Ust above: (63 > M) + (al + b2 
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+ c3>a2 + bl+cl) = (al+b3 + c3>a2 + bl+ cl). This alternative approach is variant 
1 of Approach 1 explained earlier. 

Therefore the ambiguity is implicitly resolved and deleted from the To Be Resolved list. 
(This process is performed for all the ambiguities on the To Be Resolved list.) 

The foregoing describes the invention including preferred forms thereof. Alterations and 
modifications as will be obvious to those skilled in the art are intaided to be incorporated 
within the scope hereof. 



By the authorised agents 

AJ Park 
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A CHOICE-BASED METHOD FOR CHOOSING FROM A 
RESTRICTED GROUP OF ALTERNATIVES 



FIELD OF INVENTION 

The invention relates to the field of Decision Analysis or Decision Support and more 
particiilarly to a method for choosing an alternative (the 'best' alternative) from a 
restricted group of alternatives, and/or ranking those alternatives. 



BACKGROUND TO THE INVENTION 

10 The invention is an extension of an earlier inv^tion of the applicants: "A CHOICE- 
BASED METHOD FOR CALfflRATING ADDIITVE POINTS SYSTEMS", for which 
provisional patent appUcation NZ 526647 was filed on 13 June 2003. A fiill copy of this 
patent q)pUcation is included in the Appendix to tiiis specification and is herdiy 
incorporated into the present disclosure. This earUer invention comprises a system, 

15 mettiod and computer program for determining (calibrating) the points values of additive 
pomts systems (APSs). 

The background to the invention described m tiie disclosure of the Appendix is broadly 
appUcable to the present invention. However, supplementary to the situations in which 
20 APSs are ^hcable are sitiiations in which decision makers are required to choose an 
alternative (the 'best' alternative) from a restricted group of alternatives, and/or to rank 
those alternatives. Decision makers in a wide variety of settings are regularly faced with 
having to do tiiis; examples include choosing: the best location for a building, the best 
model of car to buy, the best employee to hire or promote, etc. 

25 

Although tiie invention described in tiie Appendix is capable of choosing the *best' 
alternative from a restricted group of alternatives, and/or ranking tiiem, it is designed 
primarily for calibrating APSs in general. As such tiie Appendix describes tiie ranking of 
very large groups of alternatives, tiiereby, in effect, ranking all tiieoretically possible 
30 alternatives, mcluding alternatives that are as yet unknown or are purely hypothetical. 
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The present invention extends the invention described in the Appendix such that, instead 
of having to rank all theoretically possible alternatives, the present invention ranks a 
subset of them such that the 'best' alternative is chosen from a restricted group of 
alternatives and/or they are ranked. This serves to reduce the amount of effort required of 
decision makers in reaching their decision. 



SUMMARY OF THE INVENTION 

In broad terms, the present invention extends the system, method and computer program 
10 described in the Appendix so that the *best' alternative is chosen from a restricted group 
of alternatives, and/or they are ranked. 

The invention does this by extending the invention described in the Appendix to 
expUcitly recognise that m the process of calibrating a given APS, the restricted group of 
15 alternatives may have been ranked before the APS is fully calibrated, hi the language of 
this earUer invention, the restricted group of alternatives may have been ranked by the 
explicitly resolved ambiguities together with the inherent inequahties before the set of all 
theoretically possible profiles of the APS has been ranked. 

20 hi broad terms in one form therefore, the mvention provides a method for ranking a 
restricted group of alternatives in the context of an additive points system (APS) 
comprising a pre-determined plurality of criteria, each criterion havmg one or more 
categories wherein the pomts for each category of each criterion are determined by the 
pairwise rankmg of profile pau^, each profile comprising a set of one or more criteria, 

25 each criterion in the set instantiated with one of the categories for that criterion wherein 
the restricted group of alternatives conq>rises two or more of these sets. 

Preferably the method comprises the steps of generating ambiguous profile paira for the 
APS; simultaneously generatmg all possible ambiguities that are consistent with the 
restricted group of alternatives and storing them on a temporary list; makmg a second 
temporary Ust that initiaUy consists of the alternatives from the restricted group of 
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alternatives; solving the system of equalities/inequalities for the APS to obtain point 
values; removing ambiguities from the temporary list as the corresponding ambiguities 
for the APS are solved; removing alternatives from the second temporary list as they are 
found to be certainly ranked lower than another on the list; and returning a solution when 
the temporary list is empty or when the second temporary list contains equally ranked 
altonatives, depending on the needs of the decision mak^. 



Preferably the process of solving ambiguities for the APS may be halted once the 
10 tenq)orary list is empty. 

The invention also provides a related system and computer program implementing 
corresponding extensions to the system and computer program described in flie 
Appendix. 

15 

The invention therefore extesads the system, method and computer program disclosed in 
the Appendix by determining the exact point at which the restricted group of alternatives 
has been ranked. 

20 DETAILED DESCRIPTION OF THE PREFERRED FORMS 

This description extends the detailed description of the preferred forms of the Appendix, 
and uses the same terms and refers to the figures contained and described in tiie 
Appendix. 

25 The invention is primarily embodied in the methodology set out there both by itself and 
as implemented through computing resources such as the preferred resources set out in 
Figure 3 of the Appendix, by way of example. The mvention is also embodied in the 
software used to implement the methodology and m any system comprising a 
combination of hardware and software used to implement the methodology. 
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The present description begins with reference to Step 1 of the description of the computer 
program for implementing the method described at page 37. line 28 of the Appendix. 
Thus when the user enters the criteria and categories of the APS being calibrated (as per 
Figure 11 of the Appendix), the present invention invites him or her to also Ust the 
restricted group of alternatives in tenns of their characteristics with respect to the APS's 
criteria and categories. 

The method of the present invention will now be expired with reference to the example 
of ranking the following four alternatives in the case of an APS with just three criteria: a, 
b and c; and two categories on each: 1 and 2; such that there are sk criterion-categor^ 
variables: al, a2, 61, 62, cl and c2. By definition, the values of these variables 
monotonically increase with the categories within each criterion so that the following 
♦inherent inequalities' hold: a2 > al, b2 > bl and c2 > cl. This also is the first example of 
an APS refened to in the Appendix. 

llie four alternatives and their conesponding total score equations are (where the three- 
digit numbers are symbolic representations only): 

Alternative 1: 221 = fl2 + 62 + cl 
Alternative 2: 122 = al + 62 + c2 
Allsmative 3: 212 = a2 + 61 + c2 
Alternative 4: 21 1 = a2 + 61 + cl 

The essence of ranking tiiese four alternatives, and/or choosing tiie 'best' one. is deciding 
the point values of the six variables (al. a2. 61, 62, cl and c2) such that decision makers' 
prefeired ranking of the four alternatives (equations) is reaUsed. The system, method and 
computer program described in tiie Appendix is extended to do this as follows. 

When flie program begins to generate unresolved ambiguities for the Current Degree (as 
per Step 2 of Figure 11 of the Appendix), it also generates all the possible ambiguities 
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that are consistent with the restricted group of alternatives. These are stored on a list 
known as iho Alternatives * Ambiguities list 

The method for generating these ambiguities is the same as described on page 13, lines 
5 10 to 19 of the Appendix. In essence, it involves pairwise comparisons of alternatives and 
cancelling variables common to both alternatives. 

Thus the Alternatives ' Ambiguities list for the four alternatives in the example are: 

a2 + cl vs al + c2 

10 fe2 + cl V5ftl + c2 

a2+M vjal + M 
al -f-fc2 + c2v5a2 + M + cl 

Immediately before setting the Current Degree = 2 (Step 1 of Figure 1 1 of the Appendix) 
15 and in the Undo module, the restricted group of alternatives nominated by the user is 
copied to a list known as iho Remaining Alternatives list. 

Dming the normal processes described in the Appendix, after the user resolves the 
particular ambiguity being considered, and it is added to the Explicitly Resolved list (as 
20 per Figure 14 of the Appendix), the present invention operates as follows. 

Ambiguities on the Alternatives ' Ambiguities list are tested and removed in the same way 
as the To Be Resolved list (as per Figure 14 of the Appendix). If the Alternatives' 
Ambiguities list is not then empty, the process continues as per the process described in 
25 the Appendix; however, once iStLQ Alternatives' Ambiguities list is empty, all alternatives 
have been ranked. 

The system then solves the point values (Step 5, Figure 1 1 of the Appendix) to obtain the 
scores for the altematives, and then presents the ranked alternatives to the user. The user 
30 can contmue to calibrate the APS if he or she wishes, but the ranking of the altematives 
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wiU not change unless the user undoes some prior decisions or introduces new 
alternatives to the restricted group under consideration. 

If the Alternatives' Ambiguities list is not yet empty, a test is performed as to whether 
there is a clear 'wimier' or a set of equally ranked wimers among the Remaining 
Alternatives. To perform this second test the Remaining Alternatives Kst is processed as 
follows. 

Each alternative on the Remaining Alternatives list is conq>ared with every other to 
10 determine whether it would be mfeasible for it to be ranked below the other alternatives, 
given the Ibgylicitly Resolved Ust and the mherent inequaUties. 

With respect to Alternatives 1 and 2. for example, the system asks two hypothetical 
questions in a similar manner to Approach 2 described at page 33, Ime 10 of the 
15 Appeudix. 

Hypothetical Question 1: Does a solution exist to the system comprising the (actual) 
expUcitiy resolved inequaUties/equahties as weU as tiie mherent inequalities and the 
proposition Alternative 1 ^ Alternative 2 (here a2 + b2 + cl ^ al + b2 + c2). If the 
answer is No, and therefore it would not be theoretically possible for Alternative 1 to be 
ranked above or equal to Alternative 2, then it must be ti^e that Alternative 1 is ranked 
below Alternative 2. Hence Alternative 1 is removed from the Remaining Alternatives 



20 



Ust 



25 If instead the answer to Question 1 is Yes, and tiierefore it would be theoretically possible 
for Alternative 1 to be ranked above or equal to Alternative 2. then tiie following second 
hypothetical question is asked. 

Hypothetical Question 2: Does a solution exist to the system comprismg the (actiial) 
30 expUcitiy resolved inequaUties/equaUties as weU as the inherent inequaUties and tiie 
proposition Alternative 2 ^ Alternative 1 (here al + 62 + c2 ^ <22 + i2 + cl). If tiie 
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answer is No, and therefore it would not be theoretically possible for Alternative 2 to be 
ranked above or equal to Alternative 1, then it must be true that Alternative 2 is ranked 
below Alternative 1. Hence Alternative 2 is removed from the Remaining Alternatives 



list. 



If instead the answer to Question 2 is Yes, and therefore it would be theoretically possible 
for Alternative 1 to be ranked above or equal to Alternative 2, then the relationship 
between these two alternatives is either not yet known or they are equal. 

10 After all the alternatives on the (now possibly shorter) Remaining Alternatives list are 
tested in this manner, a final test is performed pairwise comparing each alternative on the 
Ust with every other to determine if it is possible that they are not equal, in a feshion 
analogous to the above. 

15 For example if it is possible tiiat Alternative 1 < Alternative 2, or that Altemative 2 < 
Altemative 1, (i.e., by testing wheflier it is possible that here a2 + 62 + cl<al +62 + c2, 
or al + 62 + c2 < a2 + 62 + cl) then the comparison of alternatives is abandoned and tiie 
overall process described in the Appendix continues. 



If instead all the alternatives are found to be equally ranked, or if there is only one, the 
user is presented with the 'winning' (i.e., 'best') altemative or alternatives. The user can 
continue to caUbrate the APS if he or she wishes, but the highest ranking altemative or 
alternatives will not change unless the user undoes some prior decisions or introduces 
new alternatives to the restricted group under consideration. 

The foregoing, taken together with the Appendix, describes the invention including 
preferred forms thereof Alterations and modifications as will be obvious to those skiUed 
in the art are intended to be incorporated within the scope hereof. 
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A CHOICE-BASED METHOD FOR CALIBRATING ADDITIVE 

POINTS SYSTEMS 

FIELD OF INVENTION 

The invention relates to additive points systems and in particular to a system, method and 
computer program for determining (caKbrating) the points values of additive pomts 
systems. 



BACKGROUND TO THE INVENTION 

10 Additive points systems (APSs), a type of multi-attribute utility model or multiple criteria 
decision analysis tool, are also known as 'linear', 'point-count' and 'scoring' systems. 
APSs are widely and increasingly used worldwide in most branches of medicine for 
prioritisation, diagnosis and predictive purposes and in a wide range of otiier appUcations, 
including selecting immigrants, assessing mortgage applications and predicting parole 

15 violations, business bankruptcies and coUege graduations. They are also used as a generic 
project appraisal tool. 



Canada and New Zealand, for example, recentiy developed APSs for a wide range of 
elective surgeries and other publicly-funded health cares. APSs are also used by flie 
20 immigration systems of New Zealand, Canada, AustraUa and Germany and, to a lesser 
extent, the United Kingdom. 



APSs in general represent a relatively simple solution to the pervasive problem feced by 
decision makers with multiple criteria or attributes to consider, particularly whea ranking 
alternatives or individuals. Hereinafter in this document alternatives and individuals are 
referred to generically as 'alternatives*. 

Specifically, APSs serve to combine alternatives' characteristics on multiple criteria to 
produce a single ranking of alternatives with respect to an over-arching criterion (such as, 
for example, tiie order in which to treat patients), or, more simply, to reach a decision (for 
example, whether or not to admit an immigrant). 



8J476-J 



3 



In addition to their having been near universally found in many studies to be more 
accurate than 'expert' decision makers in the respective fields to which they have been 
applied, the ^peal of points systems is that they are simple to use. 

5 

Each alternative's categorical rating on each criterion that is deemed relevant to the over- 
arching criterion is scored a particular number of points, fliat usually increase witti the 
•importance' of the categories, and the points are summed (hence additive points systems) 
to produce a total score for the alternative. 

10 

Alternatives are ranked with respect to the over-archmg criterion according to their total 
scores, including being declined altogether if a particular score 'threshold' is not reached. 
Usually the higher an alternative's score the higher its ranking, and the scores typically 
have no other meaning tiian this. 

15 

IVfost APSs for elective surgeries and immigration respectively (tiie examples referred to 
above), have between five and sevai criteria and two to five categories on each criterion. 
Figures 1 and 2 show an example of an APS used in Canada for prioritising patients for 
hip or knee replacement surgery. 

20 

In this example there are seven criteria, mostly based on types of pain and fimctional 
limitations. Each criterion has a number of mutually exclusive and exhaustive categories 
on which the consulted decision maker (usually a doctor) is asked to rate the patient being 
considered. For example, item 2 m Figure 1, refers to the criterion of 'Tain at Rest" and 
25 reqiiires that the patient be assigned to one of four categories: "None", "Mild", 
"Moderate" or "Severe". 

In general, if the criteria and the categories on each have been chosen for a particular 
APS system, then the point values for that APS must be determined (calibrated) such that 
30 the resulting ranking of alternatives represents the decision makers' preferences. The 
invention is a new sj^em, method and computer program for doing this. 
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la addition to the arbitrary assignment of points, there are two main existing approaches 
to calibrating APSs. 

5 The first regresses decision makers' judgements of the relative priorities or importance of 
a sample of real or hypothetical alternative 'profiles' (in other words, the alternative's 
categorical ratings on the criteria) on their characteristics in tenns of the criteria and 
derives point values fi-om the regression coefficients. Usually only a smaU proportion of 
existing or theoretically possible profiles is surveyed because of the responder burden on 
10 the consiilted decision makers. 

The above-mentioned decision makers' judgements are often eUcited via a visual 
analogue scale (VAS). Item 8 of Figure 2 is such a VAS, where the decisioA maker is 
asked "to rate the urgency or relative priority of this patient" between "Not urgent at all" 
15 and '«Extremely urgent (just short ot an emergency)". This 'score' and the patient's 
characteristics in terms of Items 1 to 7 of the same figure, along with analogous data for 
other patients, may then be used to calibrate the point values for the various criteria and 
categories using multiple regression techniques, as explained above. 

20 The second existing ^proach to caUbration uses decision makers' judgements of the 
pairwise relative importance of tiie APS's criteria to derive ratio scale weights. These 
wei^ts are then ^Ued to normalised criteria values to derive point values. An example 
of this type of technique is the Analytic Hierarchical Process (AJfflP). 

25 Thus the first approach assumes that decision makers' judgements have interval scale 
measurement properties and the second approach assumes tiiey have ratio scale 
properties. Botii assumptions are relatively stringent and current techniques for eUciting 
decision makers' judgements have well-known biases. Witii respect to tiie firet approach, 
for example, the validity of the dependent variable (the experts' judgments) and therefore 
tiie point values derived firom tiie estimated coefficients, can be criticised on two main 
grounds. 
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First, the scaling methods such as the VAS used to elicit the experts' judgments of the 
profiles' relative priorities are based on mere introspection rather than the ejqpression of a 
choice. Second, VAS in general may not actually have the scaling measuremcaat 
5 properties required for the valuations that they produce to be interpreted as relative 
priorities rather than just as rankings. 

It is therefore desirable to have a method of calibratmg new APSs or recalibrating or 
validating ractant ones that requires only ordinal measurement properties, specifically the 
10 positive expression of a ranking over pairs of alternatives, as this is the least stringent of 
measurranent property requirements. It would also be desirable for this method to achieve 
accurate results while reducing the burden on decision makers of ranking pairs of 
altonatives by minimising the number of pairs they have to rank. 

15 SUMMARY OF THE INVENTION 

In broad terms, in one form the invention provides a method for calibrating additive 
points systems (APSs) comprising a pre-determined plurality of criteria, each criterion 
having one or more categories wherein the points for each category of each criterion are 
determined by the pairwise rankmg of profile pairs, each profile comprising a set of one 
20 or more of the criteria, each criterion in the set instantiated with one of the categories for 
that criterion. 

Preferably the method comprises the stq)s of generating ambiguous profile pairs for the 
APS to be calibrate4 resolving the ambiguous profile pairs, and solving the resulting 
25 system of equalities/mequalities to obtain the point values. Ambiguous profile pairs are 
profile pairs in which one profile has a higher categorical rating on at least one criterion 
and a lower rating on at least one other criterion than the other profile. 

Preferably the step of generating the ambiguous profile pairs comprises the further step of 
30 removing any profiles that are theoretically impossible. 
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Preferably the step of generating the ambiguous profile pairs comprises the further step of 
reducing aU profile pairs in which both profiles have one or more of the same criteria 
instantiated with the same category. 

5 In broad terms in another form, the invention provides a system for calibrating additive 
points systems (APSs) comprising a pre-determined pluraUty of criteria for the additive 
pomts system, each criterion capable of being instantiated with one or more pre-defined 
categories; and a points calibrator configured to determine appropriate points for each 
category of each criterion by preparing data for and processing the results of the pairwise 
10 ranking of profile pairs, each profile comprising a set of one or more of the criteria, each 
criterion in the set instantiated with one of the categories for that criterion. 

Preferably the points calibrator comprises an ambiguity generator configured to generate 
ambiguous profile pairs for the APS to be calibrated. 



15 
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Preferably the points calibrator comprises a data input component configured to receive 
and store the equaUties/inequalities that result fiom resolving the ambiguous profile pairs 
generated by the ambiguity generator. 

Preferably the points calibrator comprises a solution component configured to solve the 
resulting system of equalities/inequaUties to obtain the point values. 

Preferably the ambiguity generator is further configured to remove any profiles that are 
theoretically impossible. 

Preferably the ambiguity generator is further configured to reduce all profile paire in 
which both profiles have one or more of the same criteria instantiated with the same 
category. 

In broad terms in yet another form the invention provides a conq)uter program for 
calibrating additive points systems (APSs) comprismg an initialization component 
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configured to receive and store data representing a plurality of criteria for an APS and the 
categories with which each criterion may be instantiated, an ambiguity generator 
configured to generate ambiguous profile pairs for the APS to be calibrated, a resolution 
component configured to select and present profile pairs to a user to be expUcitly 
resolved and to store the results of the resolution, an ambiguity management component 
configured to manage the resolved and unresolved ambiguities and to automatically 
resolve any ambiguities that can be resolved impUcitljr, and a solution component 
configured to solve the system of resolved inequaUties/equaUties fiwm the resolution 
component and the ambiguity management component. 

Preferably the computer program fiirther comprises a revision component configured to 
allow a user to revise any resolved ambiguities. 

Preferably the computer program is implemented using linear programming. 
BRIEF DESCBIPnON OF THE FIGURES 

Preferred forms of the method system and computer program for caUbrating additive 
points systems wiU now be described with reference to the accompanying figures in 
which: 

Figure 1 shows a prior art means of eUciting expert judgments for the purposes of 
calibrating an APS via multiple regression-based techniques; 

Figure 2 shows a continuation of the prior art means of eUciting expert judgments for the 
25 purposes of calibrating an APS via multimedia regression-based techniques; 

Figure 3 shows a preferred configuration of hardware for carrying out the invention; 



15 



20 



le m an 



Figure 4 shows a decision ti-ee identifying the 12 rankings of the 8 profiles possible 
30 exenq)lar APS with three criteria and two categories and allowing strict preferences only 
and no ties (for illustrative puiposes only); 
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Figure 5 is a flow diagram illustrating the main steps in the method and computer 
program of the invention; 

5 Figure 6 is a table showing the ambiguities from the six profiles (excluding those that are 
miambiguous) for an exemplar APS with three criteria and two categories; 

Figure 7 is a table of the total and unique ambiguities for different APSs and degrees; 

10 Figure 8 is a continuation of the table from Figure 7 of the total and unique ambiguities 
for different APSs and degrees; 

Figure 9 is a flow diagram illustrating the main components of the * efficient ambiguities 
generator* described in Step 1 of the method of the invention; 

15 

Figure 10 is a table of the sufficient (but not necessary) conditions for implicitly 
resolving the 3"*-degree ambiguities of an exemplar APS with three criteria and two 
categories; 

20 Figure 1 1 is a flow diagram illustrating an overview of Steps 1 to 5 of the computer 
program of the invention; 

Figure 12 is a flow diagram illustrating the preferred main components of the ambiguity 
generator of the computer program; 

25 

Figure 13 is a flow diagram illustrating the preferred main components involved in 
testing whether or not an ambiguity is resolved for the computer program; 

Figure 14 is a flow diagram illustrating the preferred main components involved in the 
30 explicit resolution of ambiguities for the computer program; and 
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Figure 15 is a flow diagram illustrating the preferred main components of the undo 
module (whereby expUcitly resolved ambiguities are revised) of die computer program. 

DETAILED DESCRIPTION OF THE PREFERRED FORMS 
The invention is primarily embodied in the methodology set out below both by itself and 
as implemaited through computing resources such as the preferred resources set out in 
Figure 3, by way of example. The mvention is also embodied in the software used to 
implement the methodology and in any system comprising a combination of hardware 
and software used to implement the methodology. 

In its most preferred fonn the invention is implemented on a personal computer or 
workstation operating under the control of appropriate operating and ^Ucation 
software. 



15 Figure 3 shows flie preferred system architecture of a personal computer, workstation, or 
server on which the invention could be implemented. The computer system 300 typically 
comprises a centi^ processor 302, a main memory 304. for exanqjle RAH and an 
mput/output controller 306. The computer system 300 may also comprise per^herals 
such as a keyboard 308, a pointing device 310. for exan^le a mouse, touchpad, or 
trackball, a display or screen device 3 12, a mass storage memory 314. for example a hard 
disk, floppy disk or optical disc and an output device 316 such as a printer. The system 
300 could also include a network interface caid or contixjUer 318 and/or a modem 320. 
The mdividual components of tiie system 300 could communicate tiirough a system bus 



20 



25 



The metiiod of the present invention will now be described with reference to several 
examples, begimiing witii an APS with just three criteria; a, b and c; and two categories 
on each: 1 and 2; such that there are six criterion-category variables: al, a2, b\, hi, c\ 



andc2 



30 
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For example, if this were an APS for selecting immigrants, criterion a might be 
'educational qualificaUons', b 'wealth' and c 'language proficiency'. Category 1 might be 
generically defined as 'low' and 2 as 'high'. Real immigration APSs, however, typically 
have at least twice as many criteria and categories as this simple example. 

5 

By definition, the values of these variables monotonically increase with the categories 
within each criterion so that the following 'inhwent inequalities' hold: al>a\,b2> b\ 
andc2>cl. 

10 Corresponding to all possible combinations of the two categories on the abc criteria, eight 
profiles, each with a total score equation, are represented by fliis system (where the three- 
digit numbers are symbolic representations only): 

222 = a2 + 62 + c2 
221 = a2 + b2 + c\ 
212 = a2 + bl + c2 
122 = al+fe2 + c2 
112 = aH-M + c2 
121 = al +b2 + cl 
211 = a2 + M + cl 
111 =aH-fcH-cl 

The essence of calibrating an APS such as this one is deciding the point values of the six 
variables (al, a2, hi, b2, cl and c2) such fliat decision makers' preferred (or 'valid') 
overall ranking of ttie eight profiles (equations) is realised. 

25 

The internal logic of APSs — specifically, the inviolable laws of arithmetic — restricts 
the otherwise 8! = 40,320 rankings (permutations), given strict preferences only and no 
ties (for illustrative purposes only), to the 12 rankings represented via the decision tree in 
Figure 4. The decision tree highlights the inherent contmgencies in the derivations of the 
30 profile's rankings. 

Any of the 12 rankings shown in Figure 4 can be produced from the six variables, 
depending on the values chosen for them. Ranking #1, for example, is given by al = 0, 
a2 = 4, fcl = 0, fc2 = 2, cl = 0 and c2 = 1, with the total scores: 222 = 7, 221 = 6, 212 = 5, 
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211 - 4. 122 = 3. 121 = 2, 1 12 = 1 and 1 1 1 = 0. Alternatively, ranking #12, for example, 
is given by «l=0,a2=l,M = 0, 62 = 2, cl = 0andc2 = 4, with the total scores: 222 = 
7, 122 = 6.212 = 5,112 = 4,221=3, 121 =2. 211 = 1 and 111 =0. 



vis- 
a 

X 



In general, any particular ranking of profiles is detennined by fh^ pairwise rankings 
^-vis each other. For an APS with x criteria and y categories on each, and / proffles. 

maximum of ^ pairwise rankings is possible. Thus for the exemplar APS with 

= 3 and :y = 2, and eight profiles (i.e., 2^), there is a maximum of 28 [i.e., 8(8 - l)/2] 
pairwise rankings. 

The method of the invention minimises the number of pairwise rankings that must be 
decided e^licitly (via value judgements), such that, in this example, a minimum of two 
and a maximum of four, rather than 28, is required, witii the remaining pairwise rankings 
impUaay resolved as corollaries of the expUcit rankmgs. The method of the invention 
comprises, in broad terms, three steps, as explained in tum below. The three main steps of 
the invention are illustrated in Figure 5. 

Step 1 of the method of the invention 510 involves identifying (or 'generating') the 
'ambiguities' of the APS that is being calibrated. 

Ambiguities are formed fi-om the total score equations of profile pairs whose pairwise 
rankings are a priori ambiguous, given the inherent inequaUties (as already explained. a2 
> al, bl > b\ and c2 > cl in the present example). These are profile pairs in which one 
profile has a higher categorical rating on at least one criterion and a lower rating on at 
least one other criterion than the other profile. 

For example, the pairwise ranking of profiles 221 and 212 — hereinafter referred to as 
"221 212" and corresponding to a2+b2 + cl vs a2 + bl + c2~ is ambiguous given 
a2 > al, b2 > bl and c2 > cl, as noted above. 
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On the other hand, many profUe pairs are unambiguously ranked. For example, 222 (a2 + 
62 + c2) is always pairwise ranked first and 1 1 1 (al + £>1 + cl) is always pairwise ranked 
second — e.g., a2 + bl'i- c2> al + b2 + c2 (122) and a\+b2 + c2 (122) >a\+bl + cU 
and so on. Other profile pairs are similarly unambiguously ranked, for example, al + b2 
5 +c2>al+Z?l+c2(122>112),andsoon. 



As described above, identifying and eliminating all possible unambiguous pairwise 
rankings serves to cull the rankings of the eight possible proffles fiom 8! permutations 
(40,320) to 48 (for illustrative purposes only, given strict preferences only and no ties) 
from which the 12 in Figure 4 are further culled via another aspect of the internal logic of 
APSs described below. 



15 
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The method of the invention identifies and excludes profile paira that are unambiguously 
ranked and then focuses exclusively on profile pairs that are ambiguously ranked. 

Some ambiguously ranked profile pairs can be 'reduced' by cancelling variables that are 
common to both profile equations. Thus a2 + b2 + cl vs a2 + bl + c2 (221 V5 212, as 
above) can be reduced to 62 + cl vs bl + c2 by cancellmg a2 from both profiles' 
equations. In effect, because a2 appears in both equations it has no bearing on the ranking 
of the two profiles Ihat the equations represent. We refer to such reduced forms as 
'ambiguities'. 



Moreover b2 + cl vs b\ + c2 also corresponds to 121 1 12, as al + &2 + cl vs al + £>1 + 
c2 reduces to 62 + cl vs bl + c2 after cancelling the al terms from both profiles' 
25 equations. Thus 62 + cl vs bl + c2 represents two ambiguously ranked profile pairs: 121 
vs 112 and 221 vs 212. 



However, not all ambiguously ranked profile pairs are reducible in this fashion. For 
example, no variables can be cancelled from c2 + 62 + cl vs al + 61 + c2 (221 vs 112), 
as none are common to both profiles' equations. Nonetheless, such ineducible 
ambiguously ranked profile pairs are also hereinafter referred to as ambiguities. 
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Accordingly ambiguities can be classified by the number of criteria they contain, 
hereinafter referred to as the 'degree' of the ambiguity. Thus '2'"'-degree' ambiguities 
contain two criteria, for example, bl + cl vs bl + c2, as above, and 'S^'^-degree' 
5 ambiguities contam three criteria, for example, a2 + £»2 + cl vs al+bl^- c2, as above, 
and so on. 



10 



15 



20 



30 



In general, the ambiguities for an APS with x criteria range ftom 2'^-degree to :(*-degree. 

The algorithmically simplest process for generating ambiguities is to first create all / 
combinations of the y categories on the x criteria (that is, all proffles), and thai pairwise 
rank them all against each other to identify pairs in which one profile has a higher 
categorical rating on at least one criterion and a lower rating on at least one other 

criterion. There will always be a total of pairwise rankings. 



As each ambiguously ranked profile pair is identified, it is reduced where possible by 
canceling variables in both profiles' equations, and retained only if the resulting 
ambiguity has not aheady been discovered. In other words, repUcated ambiguities are 
discarded. 



Accordingly the ambiguities for the exemplar APS with x = 3 and = 2 are set out in 
Figure 6, where the shaded ambiguities are duplicates of ones reported earlier in the table. 
In Figure 6, rankings that are unambiguous are denoted by 'n.a.'. The blank elements of 
the matrix, except for the main diagonal, are mirror images of the ones reported, and 
25 shaded ambiguities are duplicates of ones reported earlier. 

Thus it can be seen in Figure 6 that although there are nine ambiguities in total, only six 
are unique: 



(\)b2 + c\ vsb\ +c2 

(2) a2 + cl vsal +c2 

(3) ea+blvsal+b2 



81476-1 



14 

(4) a2 + b2 + clvsal+bl+ c2 

(5) 02 + 61+ c2 vs al + b2 + cl 

(6) al+b2 + c2 vs a2 + 61 + cl 

5 Clearly ambiguities (1) to (3) are 2"''-degree ambiguities (and are dupUcated in the figure) 
and ambiguities (4) to (6) are 3"*-degree ambiguities. 

In general, the number of ambiguities of each degree can be calculated from the two 
equations described below. 



10 



For a given APS with x criteria and>; categories on each criterion, the total number of 
ambiguities (including replicates) of a given degree (z) is given by: 



xl 



x(2-l-l)x(^2fc^Jx;;-- (1) 



15 The first term corresponds to 'C, (the combinations formula); the second term is the 
powers of 2 and minus one; the third term is 'C^ raised to the z*-power or, altematively, 
flie sum of the first;; - 1 natural numbers (e.g., 5(5-1 )/2 =10 = 1 + 2 + 3 + 4) raised to 
the z^-power, and the last term is a power function. 

20 Of these ambiguities, the number of unique ambiguities (excluding rephcates) of a given 
degree is given by: 

The two equations (1) and (2) differ only by the term (x) / - The meaning of this and 
25 the other three terms (common to both equations) is discussed further below when an 
alternative process for generating ambiguities is described. 

Equations (1) and (2) can be compared with Ihe equation mentioned earlier for the total 
number of pairwise rankings (includmg unambiguous ones): 

2 
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Equation (2) is paiticulaily useful because it reveals how many ambiguities at each 
degree must be resolved for any given APS. Figures 7 and 8 report the numbers of total 
and unique ambiguities for different APSs and degrees. As illustrated there, the number 
of ambiguities can be relatively large even for small values ofx and y. For example, an 
APS with jc = 6 and = 4 (twice the values for the exemplar APS above) has a total of 
2,295,756 unique ambiguities across its five degrees (2°^ to 6*). 

The sunple process described above for generating ambiguities is further sunplified if any 
profiles that are theoreticaUy impossible are culled &om the set to be pairwise ranked 
before any ambiguities are generated. 



For example, in the APS for hip or knee replacements referred to in Figures 1 and 2 it 
would be a contradiction for a patient to be rated as havmg "severe" "Fain on motion 
15 (e.g., walking, bending)" while also being rated on another of the criteria as having the 
"Ability to walk without significant pain** for a distance of "over 5 blocks". Such a 
combination of categories on these two criteria is theoretically impossible and therefore 
all profiles that mclude it could be deleted jfrom the list to be pairwise ranked. 



Sunilarly, when validating an extant APS (rather than caUbrating a new one), the profiles 
to be evaluated can be determined firom a 'stocktake' of the alternatives ranked by the 
APS over its lifetime. Any other profiles that might realisticaUy be expected in the fiiture 
could be added. 



This process also serves to increase the efficiency of Steps 2 and 3 of the method of the 
mvention described below, as in general the more profiles that are culled, the fewer 
ambiguities there are, and therefore the simpler is the cahbration exercise. 

Notwithstandmg such refinements, the simple process described above is computationaUy 
inefficient because profile comparisons that are ultimately unnecessary are perfonned and 
replicated ambiguities are discarded. 



81476-1 




16 



For example, to generate the above-mentioned 2,295,756 ambiguiUes for an APS with a: = 
6 and;/ = 4, as can be calculated from the data in Figure 8, 6,090,804 unnecessary profile 
comparisons are performed and 5,094,900 ambiguities are discarded. 

5 

The particularly preferred process for generating ambiguities according to the invention is 
therefore described below. This is also the preferred process for the computer program of 
the invention. The main con^onents of the process are illustrated in Figure 9. 

10 The particularly preferred process, hereinafter refeared to as the 'efficient ambiguities 
gaierator', is described with explicit reference to the three terms in equation (2) above 
(reproduced below) and a specific example. The example is that of generating the 3"*- 
degree ambiguities for an APS with jc = 5 and = 3, of which there are 810 in total (as 
can be calculated from equation (2) and is reported in Figure 7). 

15 

Although the equation appUes to APSs with the same number of categories on the 
criteria, as explained later below, the process can be generalised to allow the number of 
categories to vary across criteria. For example, instead of y = 3 for all 5 criteria in the 
above-mentioned APS, as is applied later, criterion a could have two categories, b three 
20 categories, c four, and so on. 



x\ 

Equation {lys first term — -— (= 'Q, the combinations fonnula) — is the number 

25 of combinations of z criteria that can be selected from x criteria. 

For the above-mentioned APS with x = 5 (that is, criteria a, b, c\ d and e), 10 
combinations of three criteria (z = 3) are possible: ahc, ahd, abe, acd, ace, ade, bed, bee, 
bde and cde. Well-known algorithms exist for generating such combinations. 
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Each combination can be thought of as fonning the 'base' for a group of ambiguities. For 
example, abc is the base for all 3"*-degree ambiguities centred on a + b + cvsa + b + c 
(hereinafter abbreviated to abc vs abc), such as a2 + 63 + c3 vs a3 + b2 + c2. 

Because the criteria in each base (10 of them in the present example) have the same 
number of categories each, the bases can be treated identically with respect to the 
following operations that correspond to tiie other two terms in equation (2). 

The second term (2"~'-l) can be intopreted as representing the number of underlying 
'structures* for a givai degree (z). Structures are generated by first listing the numbers 
between 1 and 2^' - 1 in binary form using z bits, where each bit represaits a criterion of 
two categories represented by either 0 or 1 , 

Continuing with the example of generating the 3'''-degree ambiguities for an APS with x 
=■ 5, tiie structures for z = 3 are: 

Oil 
010 
001 

These structures correspond to 3, 2, and 1 in binary form. For now, each structure can be 

thought of as being analogovis to a profile withjy = 2 for all criteria. Therefore only one 

ambiguity can be created fi-om each, which is done by finding each structure's 'Is 

complement', that is, by 'flipping the bits'; thus in the example: 

Oil vj 100 
010 101 
001 110 

Each term (either a '0' or a T) on either side of the ambig;uity structure represents a 
30 criterion, and for each criterion their relative magnitudes ('0' versus '1') represents the 
relative magnitude of the categories they represent (i.e., 'low' or 'high'), such that each 
structure has an underiying pattern. 



15 
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Thus 01 1 vs 100 in the example above (and analogously for 010 vs 101 and 001 vj 1 10) 

signifies that the first of the three criteria represented (corresponding to 0 vs 1 ) has 

a lower category on the left hand side (LHS) and a higher category on the right hand side 
(RHS) of each ambiguity that is derived from it. 

5 

Similarly, the second and third criteria (corresponding to _11 vs _00) both have higher 
categories on the LHS and lower categories on the RHS, 

The equation's third term — [^^^^^ ~ ^^^^^ ambiguities that can be 

10 geuerated from each ambiguity structure, as determined by the number of categories (y) 
on the criteria in a given base. There are three steps to generating these ambiguities, as 
follows. 



First, all of the bases are matched with aU of the structures. In the present example, the 10 

bases (abc, abd, abe, acd, ace, ade, bed, bee, bde and ode) are matched with the three 

structures (Oil 100, 010 vs 101 and 001 vs 110) to produce 10 x 3 = 30 matches: 

abc vs abc with Oil vs 100 
aba vs abc with 010 vs 101 
abc vs abc with 001 vj 1 10 
^" abd vs abd with 01 1 vs 100 

. . . and so on for another 26 matches. 

Second, the underlymg 'pattern' for each match (that is, a base with a structure) is 
implemented, according to the number of categories on the bases. 

25 

For abc vs abc with Oil vs 100, for example, with three categories on the criteria (y = 3), 

there are three ways each (that is, ^^^^ = 3(3 - l)/2 ways) of representing criterion a 

with a lower category on the LHS and a higher category on the RHS (0 vs 1 ), and 

criteria i and c both with higher categories on the LHS and lower categories on the RHS 
30 (_llvj_00): 

«2vja3 b3vsb2 c^vscl 
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al vs di b3 vs bl c3 vs cl 

al vs a2 bl vs b\ cl vs cl 

Note that — ^= -^Q, the number of combinations of categories (within the criterion) 
A* 

takoi two-at-a-tune. The output listed above embodies these combinations, where the two 
categories are simply ordered as required by the structure. 



Finally, all 3 x 3 x 3 = 27 possible combinations — i.e., \ ^^^^ == [3(3 - l)/2]' — of 

these three patterns are formed, thereby obtaining all 27 3"'-degree ambiguities 

corresponding to the match ahc vs abc with 01 1 100: 

a2 + b3 + c3 vs a3 + b2 + cl 
a2 + b3 + c3 vs a3 + b2 + cl 
a2 + b3 + c2 vs a3 + b2 + cl 
a2 + Z>3 + c3 vs a3 + bl+ c2 
a2 + b3 + c3 vs a3 + bl + cl 
a2 + b3 + c2 vs a3 + bl + cl 
a2 + b2 + c3 vs a3 + bl+ c2 
a2 + b2 + c3 vs a3 + bl + cl 
a2 + b2 + c2 vs a3 + bl + cl 
al + b3 + c3 vs a3 + b2 + c2 
al + b3 + c3 vs a3 + b2 + cl 
al + b3 + c2 vs a3 + b2 + cl 
al + b3 + c3 vs a3 + bl + c2 
al + b3 + c3 vs a3 + 61 + cl 
al + b3 + c2 vs a3 + 61 + cl 
al + b2 + c3 vs aS + 61 + c2 
al + 62 + c3 vs a3 + 61 + cl 
al + 62 + c2 vs a3 + 61 + cl 
al + 63 + c3 vs a2 + 62 + c2 
al + 63 + c3 vs a2 + 62 + cl 
al + 63 + c2 vs a2 + 62 + cl 
al + 63 + c3 vs a2 + 61 + c2 
al + 63 + c3 vs a2 + bl+ cl 
al + 63 + c2 vs a2 + 61 + cl 
al+62 + c3vja2 + 61 +c2 
al + 62 + c3 vs a2 + 61 + cl 
al + 62 + c2 vs a2 + 61 + cl 
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Analogues of the" three stqjs explained above are performed for all matches. For each of 
the 30 matches in the present example with x = 5, 3; = 3 and z = 3, 27 ambiguities 
analogous to the 27 above are generated, resulting in a total of 30 x 27 = 810 (unique) 3"*- 

degree ambiguities - i.e., x (2^-1 -1) x [^^^^J = 10 x 3 x 27 = 810. 

As noted earlier, the process outlmed above can be generaUsed to allow the number of 
categories to vary across the criteria. The key difference fiom the process explained 
above is that the underlying 'pattern' for each match (for example, abc vs abc with 01 1 vs 
100) is idiosyncratic to the criteria included, a? detennined by the numbers of categories 
on each critwion. 

For example, if in the example referred to above, iostead of three categories on all 
criteria, criterion a has two categories, b has three and c four. Thus with just two 
categories for criterion a (for 01 1 100, corresponding to 0__ vj 1__), tiiere is only one 

underlying pattern (^fclil = 2(2 - l)/2) corresponding to a lower category on the LHS 

and a higher category on the RHS of each ambiguity that is derived: 

al vsa2 



are 



As before, with three categories for criterion b (corresponding to _1_ vs _0J, there 
three underlying patterns (ifcil = 3(3 - l)/2) corresponding to a higher category on 
the LHS and a lower category on tiie RHS: 



b3 vs bl 
b3 vs bl 
bl vs bl 



Fmally, with four categories for criterion c (corresponding to I vs ^0), there are six 

underlying patterns ( i^^^ = 4(4 - l)/2) corresponding to a higher category on the 
UHS and a lower category on the RHS: 

c4 vs c3 
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c4 vs c2 
c4 vs cl 
c3 vs c2 
c3 vs cl 

^ c2 vs cl 

By taking alllx3x6=18 combinations of the above three sets of criteria-categories, all 
18 S^'-degree ambiguities corresponding to abc vs abc with Oil vs 100 (witti two, tiiree 
and four categories respectively) maybe obtained: 

10 al + b3 + c4 vs a2 + b2 + c3 

al + Z>3 + c4 vs a2 + b2 + c2 

aH- M + c4 vs a2 + 62 + cl 

al + 63 + c3 vs a2 + b2 + c2 

al + 63 + c3 vs a2 + 62 + cl 
15 al + 63+c2vja2 + 62 + cl 

al + 63 + c4 vs a2 + 61 + c3 

al + 63 + c4 vs a2 + 61 + c2 

al + 63 + c4 vs a2 + 61 + cl 

al + 63 + c3 vs a2 + 61 +c2 
20 al + 63 + c3 vs a2 + 61 + cl 

al + 63 + c2 vs a2 + 61 + cl 

al + 62 + c4 vs a2 + 61 + c3 

al + 62 + c4 vs a2 + 61 + c2 

al + 62 + c4 vj a2 + 61 + cl 
25 al + 62 + c3 vs a2 + 61 + c2 

al + 62 + c3 vs a2 + 61 + cl 

al + 62 + c2 vs a2 + 61 + cl 

This process is performed for all matches. But, because the numbers of categories for the 
30 criteria are different, each match generates an idiosyncratic set of ambiguities that 
d^ends on the numbers of categories on the included criteria. 



For example, tiie number of ambiguities generated fix)m a6c vs a6c with Oil vs 100 (as 
above) is different to the numba: from abd vs abd with 01 1 vj 100 when the numbers of 
35 categories on criteria c and are different. Therefore, in general the ambiguities for each 

x\ 

of tile (combinations of z criteria from tiie x criteria) must be generated 

individually. 
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For example, in the case of the base corresponding to the particular combination of z . 
criteria comprising the first z of the x criteria {criterion 1,. criterion 2, ... criterion z, 
where z < x), there are 2*"'-l structures as before. (The choice of the first z of the * 
criteria, mstead of any other z of the x criteria, is for notational simplicity.) Each structure 

I 2 J i 2 J ^ ••• I J ambiguities, where yi, y2 . . . yz sre Hie 

numbers of categories on each of these first z criteria. 

The total number of z*-degree ambiguities is obtained by summing the number of 

ambiguities (analogous to the example) across all bases. In general, the 

(x-z)!z! 

number of ambiguities of each degree can be calculated from, the equation described 
below. 

For a given APS with yi,y2, ... yx categories on the re^ective x criteria, the equation is 
based on the following definitions. First, Y is the set of the numbers of categories on the x 
criteria: Y= {yi, yz, ... y^} . Second, C is the set of unordered z-tuples formed by taking all 

possible —— combinations of the elements of Y, z-at-a-tibme: C = fc I c is an 

(X — Z)lz\ \ I w xo «ix 

unordered z-tuple fi-om 7, as defined above}. Each of C's elements (i.e., sets), c,, is 

numbered fiom 1 to : c, i = 1, 2, . . . ~-— . Each of c,'s elements, yy, is 

(x-z)!z! ix-z)lzl 

nimibered fi-om 1 to z: ;;fy, y = 1 , 2, . . . z. 

Applying these definitions, the number of unique ambiguities of a given degree (z), when 
the number of categories varies across oiteria, is given by: 
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Alternatively this equation can be expressed as: (2'"' - 1) x J^ifcl^ 

For common values ofyy (yy=y), that is, all criteria have the same numbers of categories, 
equation (3) is equivalent to equation (2) above. 

Either process explained above — for the same or, alternatively, different numbers of 
categories on the criteria — can be used to generate all of an APS's ambiguities of a 
given degree (such as 3«'-degree, as above), or. alternatively, ambiguities can be 
generated iadividually. 



As each ambiguity is generated it is tested for whether or not it is theoretically 
impossible, and therefore to be discarded or not, given the theoretically impossible 
profiles that were culled earUer (as described above). For tiie simple process for 
generatmg ambiguities described earUer, all theoreticaUy impossible profiles are simply 
15 removed before ambiguities are generated fi-om fbsm. 

However, because the 'efficient ambiguities generator' explained immediately above 
does not generate ambiguities firom profiles, the ambiguities must be tested as they are 
generated for whether or not the profile pairs that they represent have been culled or not 

20 

For example, for an APS with x = 4 and j = 3, the reduced form al + b3 vs a3 + b2 (i.e., 
13 — 32 — ) represoits nine ambiguously ranked profile pairs: 

1311 vj 3211 

1312 vs 3212 

2^ 1313 vs 3213 

1321 vs 3221 

1322 vs 3222 

1323 vs 3223 
1331 vs 3231 

30 1332 vs 3232 

1333 vs 3233 
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For al + 63 a3 + b2 to be theoretically impossible and therefore discardable, in all nine 
of these pairs at least one of the profiles — a miniTTunn of nine and a maximum of 18 
profiles — must be theoretically impossible. If instead at least one of the nine pairs is not 
excluded, then al + V5 a3 + Z?2 is possible and therefore ought not to be discarded. 
5 Accordingly all nine pairs must be considered before it can be determined whetiher al + 
b3 vs a3 + b2 should be discarded or not, but as soon as one profile pair is found that has 
not been excluded, the ambiguity should be retained. 

Enimierating all such profile pairs for any given ambiguity of degree z is relatively 
1 0 straight-forward, as each profile pair is based on the ambiguity in question, augmented by 
all possible combmations of the categories on the other x-z criteria. There are therefore 
y^""^ such profile pairs when the number of categories on the criteria (y) is the same. 

As noted earlier, the termy appears in equation (1) above — giving the total number 
15 of ambiguities (including replicates) of a given degree — but not in the otherwise 
idmtical equation (2) — giving the mmiber of unique ambiguities (excluding replicates) 
of a given degree. 

Accordingly ^ ^ can be interpreted as the number of *copies* of a partic\ilar ambiguity 
20 generated by the algorithmicaliy shnple process explained earlier. That is, in the example 
above with ^ = 4 and j; = 3, each of the / ^ = 3^ " ^ = 9 pairwise profile comparisons 
generates al + b3 vs a3-\- b2 (of which eight are discarded because they are replicates). 

Having generated the ambiguities. Step 2 of our method involves explicitly resolving 
25 them, one-at-a-time, while identifying all other ambiguities that are implicitly resolved as 
corollaries. This step is represented in Figure 5 at 520. 

Any of the unresolved ambiguities of any degree could be selected for explicit resolution 
during the calibration process, however there tend to be fewer ambiguities to store at 
30 lower degrees, and lower degree ambiguities are easier for decision makers to resolve, 
and so the process starts by resolving the 2°^-degree ambiguities and then proceeds to 
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resolving successively higher-degree ambiguities. If instead it were desired that the 
number of decisions be minimised at the expense of the complexity of llie decisions, the 
process should instead start by resolvmg the highest degree ambiguities and then proceed 
to resolving successively lower-degree ambiguities. 

5 

As ambiguities can only be resolved via value judgements, they must be decided by an 
individual 'decision maker' or group of 'decision makers', preferably with knowledge of 
the field in which the particular APS is to be ^pUed. For example, a group of decision 
makers for a medical APS may comprise a panel of doctors and patients. Hereinafter 
1 0 decision makers (plural) are referred to. 

Accordingly the preferences of the decision makers must be probed via a series of 
questions concerning ttieir pairwise rankings of profiles. 

15 As Usted earlier, the ambiguities for the original exemplar APS with x = 3 and = 2 are: 
(1) bl + cl vs b\ + c2. (2) a2 + cl vs al + c2, (3) a2 + bl vs al + b2, (4) a2 + b2 + cl vs 
al+bl+ c2, (5) a2 + bl+ c2 vs al+b2 + cl and (6) al+b2 + c2 vs a2 + bl + cl. 

For ambiguity (1) b2 + cl vs bl + c2, for example, the decision makers are asked, in 
20 essence: "Given two alternatives that are the same with respect to criterion a, which has 
the greater priority, _21 or _12?" 

If there is more than one decision maker, the process of getting answers to this and 
subsequent questions can be streamlined by asking decision makers to cast votes (perhaps 
25 via email) for the pairwise rankings they favour. 

However, in this type of situation the majority voting runs the risks of the well-known 
voting paradox, whereby, depending on the decision makers' individual rankings of the 
profiles, the order the ambiguities are voted on can determine the resolutions that are 
30 derived. To avoid this possibility, were it likely, the decision makers should instead be 
required to reach a consensus on their pairwise rankings. 
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Logically, three mutually exclusive and exhaustive answers to the above question are 
possible: (1) _21 is strictly preferred to _12 or (2) _12 is strictly preferred to _21 or (3) 
Ihey are equally preferred (Le., indifference between _21 and_12). 

Notationally, these three preferaices can be represented as (1) _21 > _12 or (2) _12 > 21 
or (3) _21 = _12, corresponding to (1) b2 + cl>bl+ c2 or (2) bl+c2>b2 + cl or (3) 
b2 + cl = bl+ c2 (where, as usual, is "strictly greater than" and "=» is "equal to"). 

Weak preferences, for example, _21 is at least as preferred as _12 (notationally, _21 ^ 
_12), are also a logical possibility. However strict preferences are more useful from a 
practical perspective. 

The decision makers might, quite naturally, protest that their answer to the above 
question ("Given two alternatives that are the same with respect to criterion a, which has 
the greater priority, _21 or _12?") depends on whether criterion a is rated * 1 ' or '2' (for 
both alternatives). Nonetiieless, such distinctions are precluded by the internal logic of 
APSs. If the decision makers will not answer the question as it is posed then, in effect, the 
APS itself will answer it by default, since the ambiguity v^dll eventually be implicitiy 
resolved by the other explicitly resolved ambiguities chosen by tiie decision makers. 

More specifically, for example, z/the decision makers were to decide that 121 > 112 

corresponding to b2 + cl>bl+c2 — then this impties 221 > 212, and vice versa. And 
analogously for bl+c2>b2 + cl and bl+c2 = b2 + cl. 

The key word above is "if\ as clearly neither inequality holds mtrinsically. Therefore a 
value judgement is required to resolve ambiguity (1) b2 + cl vs bl + c2: either 121 > 112 
and221 > 212 or 112 > 121 and212 > 221 or 121 = 112 and221=212. By virtue of the 
laws of aiitimietic, if one inequality or equality holds then the other must too; if one does 
not then neither does the other. It is impossible to have one half of either proposition 
without the other. 



8J476-1 




27 



Accordingly the question above could be replnased, in essence, as: "Which one of die 
following three possible rankings of two alternatives do you prefer, (1) 121 > 112 and 
221 > 212 or (2) 1 12 > 121 and 212 > 221 or (3) 121 = 1 12 and 221 = 212?" 

5 

Continuing wiih the example, suppose that in fact the decision makers resolve ambiguity 
(1) by choosing _21 > _12 (in other words 121 > 112 and 221 > 212), corresponding to 
fe2 + cl > fol + c2. Two alternative, but equivalent, approaches are available for 
identifying the implicitly resolved ambiguities. 

10 

Although these approaches — hereinafter referred to as * Approach 1 ' (of which there are 
two variants) and ^Approach 2' — differ in the means and the sequence in which the 
implicitiy resolved ambiguities are identified, both generate the same list of explicitly 
resolved ambiguities, from which the point values are derived at Step 3 described below. 

15 

However because Approach 1 becomes relatively unwieldy, and therefore more resource 
intensive to implement, for more criteria and categories than the exemplar APS with = 3 
and = 2, the particularly preferred embodiment of the computer program of the 
invention is based on Approach 2. Nonetheless, Approach 1 is described first as it is more 
20 intuitively tractable and is therefore usefiil for illustrating Approach 2, 

Approach 1 may be summarised as follows. After a given ambiguity is explicitly resolved 
by the decision makers, all other ambiguities that are implicitly resolved as corollaries are 
immediately identified by adding appropriate inherent inequalities and/or other explicitly 
25 resolved ambiguities (inequalities or equalities). Then another (unresolved) ambiguity is 
explicitly resolved by the decision makers and all its corollaries are identified. The 
process is repeated until all ambiguities have been resolved, either explicitly or implicitly. 

Thus a corollary of the explicit resolution of ambiguity (1) as M + cl > 61 + c2 (decided 
30 by the decision makers, as explained earlier) is the implicit resolution of ambiguity (4) a2 
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+ b2 + cl vs al+bl+ dZ. This is revealed by adding the inherent inequality a2 > al and 
l>2 + cl>bl+c2:ia2>ai) + ib2 + cl>bl + c2) = ia2 + b2 + cl>al+bl + c2). 

Although this addition is mathematically legitimate, its theoretical vaUdity in Ihe context 
of APSs rests on the assumption that the decision makers are logically consistent in the 
sense that their pairwise profile rankings are transitive. Transitivity means in general that 
if alternatives A>BmdB>C, then A>a 

Thus, in the present example, the e:^licit resolution of ambiguity (1) asb2 + cl>bl+c2 
corresponds to 121 > 112 and 221 > 212. as discussed earlier. Moreover 212 > 112 
because a2 > al. Therefore, assuming profile rankings are transitive. 221 > 212 and 212 
> 112 implies 221 > 112 — corresponding to a2 + 62 + cl > al + ^,1 + c2, as was 
revealed above by adding a2 > al and b2 + cl>bl+ q2. 

15 As there are no other corollaries at this point, the next ambiguity on the list, in this case 
ambiguity (2) a2 + clvsal+ c2, can be presented to the decision makers to resolve via 
an analogous question to the first one: "Given two alternatives that are the same witii 
respect to criterion b, which has the greater priority, 1_2 or 2_1?" 
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Suppose tiie decision makers answer 1_2 > 2_1, corresponding to aH- c2 > a2 + cl. A 
corollary is tiie impUcit resolution of ambiguity (6) al + b2 + c2 vs a2 + bl + cl, as 
revealed by adding inherent inequality b2> bl to al + c2> a2 + cl: (62 > bl) + (al + c2 
> fl2 + cl) = (al + 62 + c2 > a2 + 61 + cl). 

25 In addition, both (1) 62 + cl > 61 + c2 and (2) al + c2 > a2 + cl unplicitiy resolve 
ambiguity (3) a2 + 61 al + 62, as revealed by tiieir addition: (62 + cl > 61 + c2) + (al 
+ c2 > a2 + cl) = (aH- 62 > a2 + 61) (after cancelling the c terms). 



As for tiie addition of mherent inequalities and expUcitiy resolved ambiguities (as above), 
tiiis addition is also justified by tiie assumption fliat tiie pairwise profile rankings of tiie 
decision makers are transitive. 
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Thus, in the present example, the explicit resolution of ambiguity (1) as 62 + c\ > bl + c2 
corresponds to 121 > 1 12 and 221 > 212, and (2) al + c2 > a2 + cl corresponds to 1 12 > 
211 and 122 > 221. Given 121 > 112 and 112 > 211, then, by transitivity, 121 > 211 is 
5 implied. Likewise, given 122 > 221 and 221 > 212, then 122 > 212 is implied. Botii 121 
> 21 1 and 122 > 212 correspond to al + fe2 > a2 + fcl, as was revealed above by adding 
(1) M + cl > fel + c2 and (2) al + c2 > ^2 + cl. 

Thus from just two explicit decisions to resolve ambiguities (1) and (2), another three 
10 ambiguities (3, 4 and 5) are implicitly resolved, so that five of the six ambiguities are 
resolved. 

The remaining ambiguity, ambiguity (5) a2 + 61 + c2 vs al + 62 + cl, must be explicitly 
resQlved by the decision makers, via a question that is conceptually simpler than the two 
15 earlier ones: ''Which alternative has the greater priority, 212 or 121?" Suppose the 
decision makers answer 212 > 121, corresponding to a2 + 61+ c2 > al + 62 + cl. 

The system is now fully specified as: (1) 62 + cl > 61 -h c2, (2) al + c2 > a2 + cl, (3) al 
+ 62 > a2 + 61, (4) a2 + 62 4- cl > al + 61 + c2, (5) a2 + 61 + c2 > al 4- 62 + cl and (6) 
20 al + 62 + c2 > a2 + 61 + cl — as well as the inherent inequaUties a2 > al, 62 > 61 and 
c2 > cl . Of inequalities (1) to (6), only three (1, 2 and 5) were explicitly resolved by the 
decision makers, with the other three (3, 4 and 6) implicitly resolved as corollaries. 

Finally, in general but not in the present example, any explicitly resolved ambiguities that 
25 are themselves corollaries of other explicitly resolved ambiguities can be removed from 
the list from which the point values are derived at Step 3 (below). This is because only 
independent inequalities/equalities are required for deriving point values. 

A variant of the approach outlined above — in effect, its converse — is the identification 
30 of the sufficient (but not necessary) conditions for implicitly resolving ambiguities, in 
terms of (other) resolved ambiguities. Ambiguities are either explicitly resolved by the 
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decision makers or implicitly resolved when their sufficient conditions are met. Any 
ambiguities whose sufficient conditions aie not met must therefore be resolved explicidy 
by the decision makers, until all ambiguities have been resolved, either expUcitiy or 
implicitly. 



Thus in the exemplar APS with x = 3 and = 2, ambiguities (4), (5) and (6) have four 
sufficient conditions each in terms of resolved 2'«*.degree ambiguities. Specifically, 
ambiguity (4) is (implicitly) resolved as a2 + fc2 + cl > al + ftl + c2 if at least one of the 
foUowing mequaUties holds: a2 + cl > al + c2 or fl2 + cl = ol + c2 (in both cases 

because 62 > M) or £.2 + cl > M + c2 or 62 + cl = M + c2 (in both cases because a2 > 
fll). 

These and analogous sufficient (but not necessary) conditions for ambiguities (5) and (6) 
to be impUcitly resolved as a2 + fel + c2 > alH- 62 + cl and al + 62 + c2 > a2 + 61 + cl 
respectively are listed m Figure 10. No sufficient conditions exist for the opposite 
resolutions of the three ambiguities nor for equalities (that is, not for RHS > LHS nor 
LHS = RHS) m terms of resolved 2°'*-degree ambiguities. 

It is then simply a matter of comparing these conditions against resolved ambiguities (1), 
(2) and (3) (arrived at via Approach 1 explained above). Accordingly, as identified via 
shading in Figure 10, (1) 62 + cl > 61 + c2 impUcitly resolves ambiguity (4) as a2 + 62 + 
cl > al + 61+ c2, and both (2) al + c2 > a2 + cl and (3) al + 62 > a2 + 61 impUcitly 
resolve ambiguity (6) as al + 62 + c2 > a2 + 61 + cl. 

These are the same 3^'*-degree resolutions as were revealed earUer via the expUcit 
resolutions of ambiguities (1) and (2) and their additions to inherent inequalities and each 
other respectively. As such they reflect, as before, pairwise profile rankmgs that are 
transitive. 



Finally, as before, ambiguity (5) remains to be expUcitly resolved, and then the system ii 
fiilly {specified. 
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This variant of Approach 1 generalises for APSs witli higher-degree ambiguities. Their 
sufficient conditions are in terms of both individual lower-degree resolved ambiguities 
and combinations of them, of which there may be many. For example, the sufficient 
conditions for the resolution of a 5*-degree ambiguity are in terms of combinations of 
2^-degree resolved ambiguities, 3"*-degree resolved ambiguities and 4*^-degree resolved 
ambiguities. 

However, sufficient conditions for an ambiguity of a given degree can also be identified 
in terms of resolved ambiguities of the same, or even higher, degrees. In the present 
example, sufficient conditions can also be identified for resolving 2"^-degree ambiguities 
m terms of other resolved 2"^-degree ambiguities. For example, a sufficient condition for 
resolvmg ambiguity (3) al + h\ vs al + 62 as al + 62 > a2 + bl is (1) 62 + cl > 61 + c2 
and (2) al + c2 > a2 + cl, corresponding to (62 -h cl > 61 + c2) + (al + c2 > fl2 + cl), as 
explained earlier. 

This means that in general it is difficult to enumerate and check all possible sufficient 
conditions (to ensure that none are missed); therefore in practice, this variant must be 
supplemented by other methods, such as the first variant of Approach 1 explained above. 

Finally, with respect to both variants of Approach 1, because ambiguities (1) to (3) are 
not independent, both the order and the manner in which they are resolved affects the 
number of explicit value judgements that are required. The maximum number required is 
four and the minimum is two. 

For example, if inequality (3) had been decided before inequalities (1) and (2) (instead of 
after, as above), then all three ambiguities (as well as ambiguity 5) would have had to 
have been resolved explicitly. This is because mequality (3) «1 + 62 > a2 + 61 cannot be 
added to (1) 62 + cl > 61 + c2 or (2) al + c2 > a2 + cl to obtain the other uxequality, and 
yet mequalities (1) and (2) imply (3). For ambiguities (4) to (6), on the other hand, one 
and only one must be resolved explicitly. 
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AltemaUvely, for example, had ambiguity (1) been resolved as 62 + cl = 61 + c2 instead 
of 62 + cl > 61 + c2 (indifference rather than strict preference), and ambiguity (2) 
resolved as al + c2 > a2 + cl (as above), then no other expUcit resolutions would have 
been necessary. 

This can be confirmed by noting that (1) 62 + cl = 61 + c2 implies both (4) a2 + 62 + cl 
> al + 6 1 + c2 and (5) a2 + 6 1 + c2 > al + 62 + cl . In other words (62 + cl = 61 + c2) + 
(62 > 61) for both of Hiem and (2) al + c2 > a2 + cl impUes (6) al + 62 + c2 > a2 + 61 + 
cl (as above) and (62 + cl = 61 + c2) + (al + c2 > a2 + cl) impUes (3) al + 62 > a2 + 
61. Clearly, the point values derived from this system of equations and inequaUties would 
be different to the point values derived earlier. 

Similarly the system would be completely specified by the explicit resolution of 
ambiguities (1) and (6) as 61+ c2 > 62 + cl and a2 + 61 + cl > al + 62 + c2 only. Of 
particular interest is: (a2 + 61 + cl > al + 62 + c2) + (62 > 61) = (2) (a2 + cl > al + c2); 
and (a2 + 61 + cl > al + 62 + c2) + (c2 > cl) = (3) (a2 + 61 > al + 62). This illustrates 
the fact that it is not necessary to resolve lower-degree (here 2°*-degree) ambiguities 
Wore impUcitly resolving higher- degree (here 3«'-degree) ambiguities. The process can 
20 be reversed, as illustrated here. 



15 



Unfortunately, as mentioned earlier, both variants of Approach 1 become unwieldy for 
APSs with more criteria and categories than the exemplar APS with x = 3 and^' = 2. This 
is because calculating and managing all possible combinations of additions or possible 
25 sufficient conditions is resource intensive. 

The 'additions' variant of Approach 1 involves maintaining a list of aU possible 
inequalities/equalities that result fi-om the addition of each explicitiy resolved ambiguity 
witii each other expUcitly resolved ambiguity as well as with the inherent inequalities, 
30 and with every sum generated, and each of these sums with each other recursively until 
all possible additive combinations are exhausted. Each newly generated ambiguity is 
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^checked off against the list of implicitly resolved ambiguities: if it is not on the list then 
it is yel to be resolved. 

Similarly, the ^sufficient conditions' variant of Approach 1 involves managing a Ust of all 
5 implicitly resolved ambiguities, identifying all possible sufficient conditions of lower 
degrees, of which there may be a great many combinations, and identifying ambiguities 
implied by the same or higher degree explicitly resolved ambiguities (by some other 
means). 

10 Approach 2, on the other hand, which is explained below, is more efficient and is 
therefore the preferred method used in the computer program of the invention. 

In summary. Approach 2 involves testing ambiguities individually for whether or not they 
are implicitly resolved as corollaries of the explicitly resolved ambiguities (that were 
15 resolved earlier). If a given ambiguity is identified as having been implicitly resolved 
then it is deleted. If instead it is not implicitly resolved then it must be explicitly resolved 
by the decision makers. The process is repeated until all ambiguities have been identified 
as having been implicitly resolved or they are explicitly resolved. 

20 Thus, with refermce to the exemplar APS with x = 3 and y =2, after the decision makers 
resolve ambiguity (1) as Z?2 + cl > Z^l + c2 as described earlier, the next ambiguity on the 
Ust ((2) al + c2 a2 + cl) is tested for whether or not it is implicitly resolved as a 
corollary of i?2 + cl > 61 + c2, as well as the inherent inequalities a2 > al, b2 > bl and 
c2>cl. 

25 

Here, as in the computer program of the invention, this and subsequent tests can be 
performed via linear programming. In effect, this test is performed by asking the 
following two hypothetical questions (of the method, not the decision makers). 

30 Hypothetical Question 1: If it were the case that the ambiguity in question [here (2) al + 
c2 vs a2 + cl] had been implicitly resolved as LHS > RHS (i.e., al + c2 > a2 + cl), then 
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does a solution exist to the system comprising this hypothetical inequality and the (actual) 
explicitly resolved inequalities/equalities [here (1) bl + c\ > 61 + c2] — as weU as the 
inherent inequalities [here al > al, b2 > bl and c2 > cl]? (Yes or No?) 

Ifihe answer is No — and therefore it would not be theoretically possible for the decision 
makers, if they wanted to, to explicitly resolve the ambiguity in question as LHS > RHS 
(here al + c2 > a2 + cl) — then it must be true that either RHS > LHS or LHS = RHS 
(i.e., either a2 + cl > al + c2 or al + c2 = a2 + cl). This impUes that the ambiguity in 
question (here ambiguity 2) has been impUcitiy resolved (Le., as either a2 + cl > al + c2 
or al + c2 = a2 + cl). Hence it is of no further use and can be deleted. 

If instead the answer to Question 1 is 7es — and therefore it would be liieoretically 
possible for the decision makers, if they wanted to, to expUcitly decide LHS > RHS (here 
al + c2 > a2 + cl) — flien the following second hypothetical question is asked. 



Hypothetical Question 2: If it were instead the case that the ambiguity in question (here 
ambiguity 2) had been impUcitly resolved as RHS > LHS (that is, a2 + cl > al + c2), 
then does a solution exist to the system comprismg this hypothetical inequality and the 
(actual) explicitly resolved inequaUties/equalities [here (1) 62 + cl > 61+ c2, as before] 
20 — as well as the inherent inequaUties [here a2 > al, 62 > 61 and c2 > cl]? (Jes or No!) 

If±& answer is JVb — and therefore it would not be theoretically possible for the decision 
makers, if they wanted to, to explicitly resolve the ambiguity in question as RHS > LHS 
(here a2 + cl > al + c2) — then it must be true that either LHS > RHS or LHS = RHS 
25 (either al + c2 > a2 + cl or al + c2 = a2 + cl). This implies that the ambiguity in 
question (here ambiguity 2) has been implicitly resolved, in this case as al + c2 > a2 + cl 
or al + c2 = a2 + cl . Hence it is of no further use and can be discarded. 



ifinstead the answer to Question 2 is Yes then it must be uiferred that as well as it being 
theoretically possible for the decision makers, if they wanted to, to explicitly resolve the 
ambiguity in question as LHS > RHS (here al + c2 > a2 + cl) from Question 1 that it is 
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also theoretically possible for them to resolve it as RHS > LHS {a2 + cl > al + c2). This 
implies that the ambiguity in question (here ambiguity 2) is not impliciUy resolved as a 
corollary of the explicitly resolved ambiguities, and therefore it must be expUcitiy 
resolved by the decision makers. 

In the case of ambiguity (2) al + c2 vs a2 + cl the answers to Questions 1 and 2 are Yes 
and Yes, and so the ambiguity should be presented to the decision makers for them to 
expUcitly resolve. As for the earlier demonstration of Approach 1, suppose it is decided 
al + c2>a2 + cl. 

The next ambiguity on the Kst — (3) fl2 + fcl V5 aH- M — is then tested via the same 
process as outlined above. This time, though, the list of (actual) explicitly resolved 
inequaUties/equaUties against which (3) al + b\ > a\ + hi (i.e., LHS > RHS. as for 
Question 1) and then, if necessary, (3) al + M > a2 + fcl (RHS > LHS, as for Question 2) 
are tested comprises (2) al + c2 > a2 + cl as well as (1) bl + c\>h\^ cl (as before). 



Thus the list of explicitly resolved inequalities/equaUties is continually updated — 
including in general but not in the present example, as for Approach 1, the identification 
of any explicitly resolved inequaHties/equalities that are themselves coroUaries of others 
20 on the list. 

The process is repeated for all ambiguities untU all of them have been identified as having 
been implicitly resolved or they are explicitly resolved by the decision makers. This can 
be summarised for the present example as follows. 

25 

For ambiguity (3) the answer to Question 1 is No, and therefore this ambiguity is 
identified as having been implicitly resolved as a corollary of the explicitly resolved 
ambiguities. For ambiguity (4), the answer to Question 1 is Yes but the answer to 
Question 2 is iVb, and therefore this ambiguity is identified as having been implicitiy 
30 resolved. 
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For ambiguity (5) the answers to both questions are Yes, implying that this ambiguity has 
not been implicitly resolved, and so it must be expUcitly resolved by the decision makers: 
as a2 + 61 + c2 > al + fc2 + cl, as for the demonstration of Approach 1. Finally, for 
ambiguity (6) the answers are Yes and No, and therefore this ambiguity is also identified 
5 as having been implicitly resolved. 

Thus, of the six ambiguities, three had to be expUcitly resolved by the decision makers 

(1) 62 + cl > il + c2, (2) al + c2 > a2 + cl and (5) a2 + bl + c2> al + b2 + cl (the 
same three as for Approach 1) — with the other three (3, 4 and 6) identified as having 
10 been implicitly resolved as corollaries. 

The final step of Ihe method (Step 3) of the mvention involves simultaneously solving the 
system of (independent) e7q>Ucitly resolved ambiguities (mequalities and equaUties) and 
mherent inequalities to obtain the point values. This step of the method is i^resented in 
15 Figures at 530. 

As described above, any explicitly resolved ambiguities that are themselves coroUaries of 
other expUcitly decided inequaUties are removed, because only independent 
inequaUties/equaUties are required for deriving the point values, 

20 

For the exemplar APS with jc = 3 and = 2, one solution for (1) b2 + cl>bl+ c2, (2) al 
+ c2 > fl2 + cl and (5) a2 + bl + c2> al + b2 + cl and a2 > al, b2 > bl and c2 > cl is: 
al =0,a2 = 2, bl = 0, 62 = 4, cl = 0 and c2 = 3. These point values produce ranking #9 
in Figure 4. 

25 

The method explained above may be summarised as foUows. First (Step 1), generate the 
ambiguities of the APS that is to be calibrated as shown at 510 in Figure 5. 

Then (Step 2) explicitly resolve them via the value judgements of the consulted decision 
30 makers, while identifying all other ambiguities that are implicitly resolved as coroUaries, 
until aU ambiguities are resolved. This stq> is Ulustrated at 520 in Figure 5. Although, in 
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theory, the implicitly resolved ambiguities can be identified vi^ Approach 1 or Approach 
2 (botli explained above), the latter is more efficient and therefore it is the particularly 
preferred method for the computer program of the invention, as explained below. 

Finally (Step 3), simultaneously solve the system of expUcitly resolved ambiguities (that 
is, inequalities and equalities) and inherent inequaUties to obtain the point values for the 
APS. This step is illustrated at 530 in Figure 5. 



The property (assumption) that the profile rankings of the decision makers are transitive 
10 (or logically consistent), enables the number of ambiguities that must be expUcitly * 
resolved at Step 2 to be nunimised, with the remainder emerging impUcitly as corollaries. 

For APSs with more criteria and categories than the exemplar with jc = 3 and = 2, 
minimising the number of explicitly resolved ambiguities, and accordingly the number of 
15 value judgements required fix>m the decision makers, is a significant practical advantage 
of our method. 



The computer progrmn for implementing the method described above wiU now be 
described more particularly The computer pix>gTam of the invention comprises, in broad 
20 terms, five steps, as explained in turn below. An overview of the five steps is illustrated 
inFigurell. 



Before starting the program, the user — who may be one of the decision makers 
consulted to resolve the ambiguities, or, altematively, he or she may be a faciUtator of the 
25 calibration process ~ must have chosen the criteria and categories of the APS that is to 
be calibrated and ranked each criterion's categories. 

At Step 1, when the program begins, the user may be asked to enter a tifle for the APS 
and the criteria and their categories, and to rank the categories for each criterion. The 
30 criteria and categories must be labelled in terms of the variables (e.g. 'a\\ 'al' 'a3\ etc.) 
and verbally described in preparation for their later presentation to the decision makers. 



8J476-J 



38 



The user is given the opportunity of hsting theoretically impossible combinations of 
criteria and categories that partially or fully specify profiles — known as the To Be 
Excluded list — and that are therefore to be used to cull ambiguities that are generated by 
5 the program. This list may be left empty if the user wishes. 

After the program is initialised, it calculates the number of unique ambiguities to be 
resolved using equation (3) above. This can be rq)orted to the user and used to estimate 
whether the system can be solved in an acceptable amount of time given the computing 
1 0 resources that are available. 

Step 2 of the program involves generating the ambiguities using the 'eflBciCTLt ambiguities 
generator' described earlier. An overview of this step is illustrated in Figure 12. 

15 For simplicity and efficiency, ambiguities are generated one degree at a time, beginning 
with the 2°^-degree, as determined by the value of a control variable — known as the 
Current Degree variable — which is initially set to 2. 

As each ambiguity is generated it is checked for whether or not it is theoretically 
20 possible, given the To Be Excluded list of partially or fully specified profiles. If the 
ambiguity is theoretically possible it is then tested via Approach 2 of Step 2 of the 
method of the invention explained earlier for whether or not it is implicitly resolved by 
the explicitly resolved ambiguities. An overview of the procedure is illustrated in Figure 
13. 

25 

This and similar tests may be performed via linear progranmiing, which is described 
below. 

If an ambiguity is found to be implicitly resolved then it is discarded; otherwise it is 
30 added to a list known as the To Be Resolved list. (Note that when the 2°^-degree 
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ambiguities are generated, none wiU be impUcitly resolved because none have yet been 
explicitly resolved.) 

Step 3 of the program is to present the To Be Resolved list of unresolved ambiguities for 
5 the current degree to the user. An overview of this step and Step 4 (explained below) is 
illxistrated in Figure 14. 

The user can choose to view the ambiguities either in equation form (for example, a\ + 

b2 vs a4 + bl) or symbolically (for exanq)le, 21 or 41 ), which are ordered 

10 either randomly or by their criteria and categories. The user is invited to select an 
ambiguity for the purpose of expUcitly resolving it. The selected ambiguity is 'translated' 
verbally in teams of the criteria and category descriptions. 



15 



20 



25 



If the user desires, he or she may skip this particular ambiguity and select anoflier one, or 
she may resolve it by clicking one of three buttons labelled (in essence): "LHS greater" or 
"RHS greatei" (i.e., < or RHS preferred) or «LHS and RHS equal". If "BHS greater 
(prefenred)" is chosen, for consistency, the LHS and RHS of the resolved ambiguity are 
switched and stored as RHS > LHS. 

The system can also permit weak inequalities (such as "LHS greater than or equal to" and 
"RHS greater than or equal to"), however this means that some ambiguities will later be 
partially (weakly) solved and so some buttons in the user interface must be disabled when 
the ambiguity is selected. Note that in any resulting APS the result will be either "greater 
than" or it will be "equal to" but it wiU not be both. The system can also be designed so 
that only strong mequalities and no equalities are permitted, thereby producing strict 
profile rankings only. 



Step 4 is for the program to remove the explicitly resolved ambiguity (as above) from the 
To Be Resolved Ust and add it — as either an inequaUty or equality (depending on how 
30 the ambiguity was resolved) — to the Ust of expUcitly resolved ambiguities, known as the 
Explicitly Resolved Ust. An overview of this step and Step 3 is Ulustrated as Figure 14. 
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Each mequality/equaliiy on the Explicitly Resolved list is tlien tested as to whether or not 
it is implied by the others on the list. Any found to be implied may be marked as being 
'redundant' and hereinafter ignored, but not deleted because they may be re-used later if 
5 any explicitly resolved ambiguities are later revised by the decision makers (explained 
below). 

All ambiguities on the To Be Resolved list are then tested for whether or not their 
resolution is implied by the (non-redundant) inequalities/equalities on the Explicitly 
10 Resolved list, and if so they are deleted from the To Be Resolved Ust This test was 
e35)lained in earlier in temis of Hypothetical Questions 1 and 2 of Approach 2 of the 
method's Step 2 and is illustrated in Figure 13. 

In essence, the test involves finding whether or not a solution (in terms of feasible point 
15 values) exists to a system comprising the explicitly resolved inequalities/equalities and 
inherent inequalities, and each of the possible hypothetical inequalities in turn (that is 
LHS > RHS and RHS > LHS) corresponding to the ambiguity in question. This and the 
earlier tests based on determining the existence of solutions are performed via linear 
programming (LP) with inequality and equality constraints. LP is discussed in more detail 
20 in the final section below. 

If the To Be Resolved hst is not empty, the program returns to Step 3. If instead the To Be 
Resolved list is empty and the Current Degree is equal to the number of criteria (jc) in the 
particular APS being calibrated, the program proceeds to Step 5. Altematively, if the 
25 Current Degree is less than the APS's number of criteria (and the To Be Resolved list is 
empty), the Current Degree is increased by * T and the program retums to Step 2. 

The user may choose at any time to view the explicitly resolved ambiguities for all 
degrees and 'undo' any of them. Explicitly resolved ambiguities that are 'redundant* are 
30 displayed in a different colour to alert the user to the fact that undoing them alone will 
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have no material effect. An overview of the *undo module', as referred to in Figure 1 1, is 
illustrated in Figure 15. 

When explicitly resolved ambiguities are undone, they are deleted from the Explicitly 
Resolved list. If any non-redimdant explicitly resolved ambiguities are undone, the 
expUcitly resolved ambiguities marked redundant are re-tested for whether or not they are 
still redundant, and marked accordingly. 

Also, if non-redundant ambiguities are undone, the Current Degree variable is reset to the 
degree of the lowest-degree ambiguity that was undone, and the program restarts from the 
second step in the computer program as set out above. 

When all the ambiguities for all degrees have been resolved, the list of explicitly resolved 
inequalities/equalities and inherent inequalities is solved via linear programming for the 
point values of the APS. These are presented to users, as well as a range of 'sunomary 
statistics', including the numbers of ambiguities and explicitly resolved ambiguities at 
each degree and, it the user wants them, the inequalities/equalities that were chosen. 

In the interests of deriving point values that are integers and as low as possible, thereby 
maximising their 'user-fiiendliness', integer programming and an objective function that 
minimises the sum of the variables (for example, al + a2 + a3 + fel + 62 + W + cl + c2 + 
c3, etc.) may be specified. 

Altematively, were it desired by users that the point values be normalised to a particular 
range for the profile total scores, such as 0 to 100, the derived point values are scaled 
accordingly. However this usually forces some point values to be fractions, which is not 
as 'user-fiiendly' as integers. Accordingly both integer and normalised values can be 
calculated and presented. 

As explained above, linear programming (LP) may be used in the computer program of 
the invention. Specifically, LP may be used at Steps 2 and 4 to determine whether the 
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resolution of an ambiguity is implied by the inequalities/equalities on the Explicitly 
Resolved list. In Step 5, LP is used for deriving the point values v/hen all ambiguities 
have been resolved. The foUowmg features must be observed when LP is used. 

5 First, the inequalities/equalities and' inherent mequalities must be converted to a form 
suitable for LP. For example, the mherent inequalities di>a2> a\ must be written as a2 
- al = 1 and oB - a2 = 1, and the explicitiy resolved ambiguities fl2 + 62 = a3 + b\ and 

fl2 + fe3>a3 + fc2asfl2 + 62-a3-M=0anda2 + M-fl3-62=L Setting tiieRHSof 
the weak inequality to "1" (tiiat is, 'epsilon') corresponds to the initial inequality bemg 
10 strict, altiiough other values for epsilon may perform at least as well. 

Second, in Steps 2 and 4, tiie LP objective function need only be "0", as aU that is bemg 
tested for is the existence of a solution rather tiian a particular optimal solution. 

15 Finally, because tiie variables corresponding to the lowest category on tiie respective 
criteria (al, b\, cl, and so on) are, effectively, 'numeraires' or 'baseline' values for flie 
respective criteria, ttiey can be set equal to zero, thereby eliminating them from the 
system to be solved and mcreasing the efficiency of the LP algorithm. 

20 Below is an illustration of how LP may be used to test whetiier or not an ambiguity on the 
To Be Resolved list is impUed by tiie inequalities/equalities on the Explicitly Resolved hst. 

For the sake of tiie example, suppose tiie ambiguity m question is alH- 53 + c3 vs al + h\ 
+ cl and tiie following inequalities/equalities are on tiie Explicitly Resolved Ust (where 
tiie shaded inequality signifies tiiat it is 'redundant' in tiie sense, as discussed earUer, tiiat 
it is implied by others on the list). 

a2 + Z>2 = a3 + b\ 
a2 + b3 >a3 + b2 

a2 + cl> al + c3 
Z)l + c2 = W + cl 
M+c3=M+c2 
al+fc2 + c3>a2 + fcl+cl 
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In addition, the inherent inequalities of tliis exemplar APS with jc = 3 and>; = 3 aie: 

a3>a2> al 
b3>b2> bl 

5 c3>c2>cl 

These inequalities/equalities (and ignoring the shaded redundant inequality) — and given 

al = 61 = cl = 0 (as discussed above) — are represented in a form suitable for LP, as 

follows. By definition, a solution (in terms of the point values) exists to the LP problem: 

minimise 0 
subject to: a2 + 62-a3 = 0 

a2 + 63-a3~fe2> 1 

c2-b3^0 

15 c3-M-c2 = 0 

fc2 + c3-a2S:l 
a2>l 
a3-a2> 1 
b2>l 

20 b3-b2^1 

c2>\ 
c3 - c2 > 1 

To test whether al + &3 + c3 vs a2 + 61 + cl is implied by the inequalities/equalities on 
25 the Explicitly Resolved Ust, the above LP problem is first augmented with b3^ c3-a2> 
1 (i.e., al + 63 + c3 > a2 4- 61 + cl), and tested for whether or not a solution to this new 
problem exists. 

A solution does exist, and so the original LP problem is next augmented with a2 - 63 - 
30 c3 > 1 (i.e., a2 + 61 + cl > al + 63 + c3) and tested for whether or not a solution to this 
second new problem exists. 

In this case there is no solution, and so it must be inferred that either al + 63 + c3 > a2 + 
61 + cl or al + 63 + c3 = a2 + 61 + cl is implied by the inequalities/equalities on the 
35 Explicitly Resolved list. The first inequality above is easily confirmed here by adding 63 
> 62 to the last explicitly resolved ambiguity on the first list above: (63 > 62) + (al + 62 
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+ c3 > a2 + i>l + cl) = (al + b3 + c3 > a2 + bl + cl). This alternative ^proach is variant 
1 of Approach 1 explained earlier. 

Therefore the ambiguity is hnplicitiy resolved and deleted from the To Be Resolved list. 
(This process is performed for all the ambiguities on the To Be Resolved list.) 

The foregoing describes the invention including preferred forms thereof. Alterations and 
modifications as will be obvious to those skilled in the art are intended to be incoipoiated 
within the scope hereof. 



By the authorised agents 

AJ Park . — - 
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5« Abnorxnal findmas on physical exam rented to affected joint (e.g. deformity, mslabllity, leg length 
djffi^ace, restriction of range of motion on examination]: 

□ Nous/Mild 
n MIcderate 

□ Severn 

S, PotfinlSal for progression of dbease documented fey radiographfc findmgs (e-g-recurienl dislocation, 
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{OrasAr a line across ifie scale) 



tiot Urgent at; all Extreme^ Urgent 

(fusitshortof an emergency) 
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(5) a2 + b\+ c2 vs al + b2 + cl 


bl+c2>b2 + cl 
or fcl + c2 > 62 + cl 
ora2 + M = al +62 
ora2 + 61=al + 62 


(6) al + b2 + c2 vs a2 + bl+ cl 


or al + c2 = a2 + cl 
or al + 62 = a2 + 61 
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